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I. BACKGROUND 



ORIGIN OF THE STUDY 

T he Texas Successful Schools Study: Quality Education for Limited English Proficient 
Students (Study) was realized through the collaborative efforts of the Texas Education 
Agency (TEA), the Charles A. Dana Center at the University Texas in Austin, Texas, the 
Texas A&M University~Coipus Christi, and the seven elementary school campuses comprising the 
Study cohort. The Study was conducted pursuant to a recommendation made in A Report to the 75^ 
Texas Legislature fi'om the Texas Education Agencv-December 1996. 

The Study evolved fi'om priority goals listed in the TEA’s Sti'ategic Plan of December 1996, as a 
leadership effort to improve the Texas public educational system. The Study will sei-ve to build the 
capacity of school campuses by reporting on local excellence and achievement accomplished by 
the seven successful schools. The inforaiation acquired through the Study effort has been further 
elaborated in this Technical Manual in an attempt to provide assistance to all school districts, so 
that all students may experience academic success and meet the State’s standards. 



NEED FOR THE STUDY 

The new student enrollment in Texas public schools continues to present significant multiple 
challenges to the Texas Education Agency (TEA), the State Boai'd of Education, the Texas Legislature, 
local school boards, administrators, teachers, and the community-at-large. These challenges become 
more pronounced when consideration is given to the demographic characteristics, e.g., numbers, 
ethnicity and program-type of the new student enrollment. 

According to TEA data contained in the Public Education Inforaiation Management System (PEIMS), 
the total state enrollment in Texas public schools for the four-year period analyzed for the Study 
increased fi'om 3,601,834 in 1993-94, to 3,891,877 in 1997-98. These figures represented an increase 
of 290,038 new students in PreKindergarten through Grade Twelve (PreK- 12). 

In 1 997-98, Texas public schools reported 5 1 9,92 1 students enrolled and identified as LEP in Early 
Education (EE) through Grade Twelve ( 1 2). An analysis for a six-year period between 1 99 1 -92 and 
1997-98 indicates that Texas public schools experienced an increase of 44 percent in the LEP 
population. The enrollment figures for each of the 20 education seiwice centers in the state indicate 
that approximately 85 percent of all new LEP student enrollment was evident in six of the 20 
service center areas. These areas included; Edinburg in the lower Rio Grande Valley, Houston, 
Richardson (including the Dallas Metroplex), Ft Worth, El Paso, and San Antonio. Edinburg and El 
Paso were the only sei-vice center areas with school distiicts along the United States/Mexico border. 
Although the increase in enrollment indicates an overall gi'owth of 44 percent in the LEP population 
in the state over a six-year period, it is important to note that 1 22,526 or 77 percent of the 1 58,794 
new LEP students reported were enrolled in the elementary grades in 1997-98. 
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According to infoimation available from the State Board for Educator Certification (SBEC) for 
1996-97, approximately 95 percent of the total number of teachers assigned to non-bilingual 
classrooms in Grades 1-6 were certified for the assignment, whereas only 59 percent of the total 
number of teachers assigned to bilingual classrooms in Grades 1-6 were certified. The remaining 
4 1 percent of teachers, not properly certified, were also products of formal teacher-tiaining programs, 
such as the college preparation progiam and the Alternative Certification progiam. 

The Study was conducted by TEA in an effort to assist school districts faced with a continued 
increase in enrollment, and challenged by a high incidence of LEP students and teacher shortages. 
Although executive management at TEA recognized that a study would not result in a remedy to 
eveiy situation that school disti icts have to address, it was evident that TEA had to provide focused 
leadership efforts to assist school administiatprs and policy-makers in their dedicated efforts to 
meet these challenges. In accepting a shared responsibility, and in keeping with the mission of 
TEA, the Study was undertaken as a first in a series of studies to focus attention on the education of 
students with special needs. 



SCOPE OF THE STUDY 

The Study was assigned to the Program Evaluation Unit in the Office for the Education of Special 
Populations to examine the significant features of successful school programs for limited English 
proficient (LEP) students, as evidenced by test results on the Texas Assessment of Academic Skills 
(TAAS) for a five-year period. Based on overall student, i.e., LEP, foraier LEP and Non-LEP 
perfomiance on the TAAS tests, the Study sites were selected on having a rating as either 
“Recognized” or “Exemplaiy” schools by the Agency’s School Accountability System for each of 
three yeare. These included the 1995-96, 1996-97 and 1997-98 school years. In addition to the 
accountability ratings mentioned, the seven Study sites had to meet established criteria that included 
the following school characteristics: 

❖ Schools enrolled more than 40 percent LEP students during the 1996-97 school year 

*** Schools enrolled more than 50 percent economically disadvantaged students during the 1996- 
97 school year, and 

❖ School had zero TAAS exemptions during the 1996-97 school year 

The Study was also designed to address specific research questions that delved into demographics, 
effective practices in use, characteristics of the seven Study sites, and the educational background 
and experience of educational personnel assigned to the LEP population in Grades PreK-5. The 
Study, which was conducted over a 24-month period as part of the Commissioner’s Educational 
Research Initiative, included a review of research studies, journals and public policy documents. 
The review of the literature regarding the education of language minority children allowed for 
obseiwations regarding the relationship of the instmctional and assessment practices in the seven 
successful schools to theory found in the literature. The entire Study effort was focused on the 
following successful schools: 
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SUCCESSFUL SCHOOLS 



Bowie Elementary 


PhaiT San Juan Alamo ISD 


Alamo, Texas 


Clover Elementary 


Phan- San Juan Alamo ISD 


San Juan, Texas 


Castaneda Elementary 


Brownsville ISD 


Brownsville, Texas 


La Encantada Elementary 


Sah Benito CISD 


San Benito, Texas 


Kelly Elementary 


Hidalgo ISD 


Hidalgo, Texas 


Scott Elementary 


Roma ISD 


Roma, Texas 


Campesti-e Elementaiy 


SocoiTO ISD 


El Paso, Texas 
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II. GENERAL DESIGN OF THE STUDY 



STUDY DESIGN 

The Study essentially employed the unique framework of a Multiple Operations Descriptive 
Longitudinal [MODE] design, and frequently references the MODE design basically as “multiple 
operations” in the Study document. This framework differs significantly from traditional quasi- 
experimental designs proposed in the classic sense by Cook and Campbell (1979), among others. A 
number of reasons for adopting the MODE design were as follows: 

❖ The Study did not look at causality, prediction, or the testing of hypotheses, which are 
characteristics of the traditional paradigm. The multiple operations characteristics of the design 
provided a widened scope to describe outcomes and identify practices, which were aligned 
with the scope of the Study 

*** The adopted design allowed for the creation of several comparison groups, e.g., target, external 
campus, and peer campus groups, within the boundaries of a framework with descriptive 
chaiacteristics. The fomiation of these Study groups differed in underlying assumptions oftiaditional 
control groups as may be treated in the Unti'eated Contiol Group Pretest and Posttest Design. The 
MODE pai-adigm allowed the use of comparable gioups more along the lines of a number of realistic 
and similai' demographic features, without uniealistic statistical controls. These conti"ols oftentimes 
pose threats to validities and reliabilities within the ti"aditional research paradigms 

♦J* The MODE approach focused on why the outcomes came about 

❖ The focus on describing outcomes through a multiple operations approach was predicated on 
various theories, sources of data, and methodologies among other considerations 

❖ The flexibility of the design allowed for the descriptive longitudinal comparisons of academic 
outcomes for two unique cohorts of students drawn from the Study campuses and external 
campuses 

❖ The MODE design employed both qualitative and quantitative approaches to illustrate both 
outcomes and effective practices 



METHODOLOGY 

The methodology used for the Study is a multiple operations framework that presents, and as 
appropriate, clarifies data outcomes. When more than one method is used, there is greater potential 
for credibility in comparing and confimiing findings. The Study approaches to data collecting and 
analyses of data were both qualitative and quantitative. The qualitative approach to data is typically 
used to answer questions about the nature of the phenomena with the puipose of describing the 
phenomena for understanding from the participants’ point of view (Lincoln & Cuba, 1989; Patton, 
1980). From a quantitative perspective, the Study utilized descriptive statistical application including 
counts, percentages, various gi aphic and tabular displays, as well as measures of central tendency 
and variability. 
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In discerning the significant features of successful schools, the Study employed descriptive methods 
and incoiporated mixed methodology and multiple operations approaches (Tashakkori and Tedlie, 
1 998; Denzin and Lincoln, 1994;Ragin, 1990; Webb, Campbell, et. al., 1965). Within the framework 
of the multiple operations design, these descriptive methods do not necessarily seek or explain 
relationships, test hypothesis, or make predictions. Research aimed at these more powerful purposes 
may incoiporate descriptive methods. The uniqueness of these approaches are in line with the 
recommendation from the National Academy of Education (1999, p.l 1), calling, “for new forms of 
research organization that are focused on practice and on engaging researchers and practitioners 
together in problem solving and theoretical analysis.” 



APPLICATIONS OF THE DESIGN 

The MODE design employed by the Study allowed applications of multiple theories, multiple 

methods, multiple investigators, and multiple sources of data, as described below: 

♦♦♦ multiple theories'. In addition to considering options within the arena of research design, the 
MODE design also allowed for reflection on multiple theories discerned from the review of the 
literature on educating students with special needs. This characteristic of a multiple operations 
design enhances the overall direction of a study by considering parts of alternative explanations, 
multidisciplinary considerations, and views of subject populations, outcomes, and practices 
which are external to prevailing paradigms (Deslauriers, 1991). The employment of multiple 
theories brings to bear the concept of Mutual Simultaneous Shaping (Lincoln and Cuba, 1985) 
where “eveiything influences eveiything else,” replacing the empirical reductionism of causality 

multiple methods: One of the applications of multiple operations designs has been the 
employment of various quantitative and qualitative methods. These multiple methods in turn 
contiibute to the validity of the design and provide for a triangulation of outcomes (Lee, 1991; 
Gable, 1994). The methods employed by the Study included features of both naturalistic and 
quasi -empirical models. The multiple operations approach was then able to employ comparison 
gi'oups without random assignment, and still be able to convey depictions regarding the “why” 
and “how” of observed phenomena for the Study sites, individually and collectively. Since each 
methodology by itself had its own inherent stiengths and weaknesses, the variety of approaches 
built upon the collective effectiveness, while minimizing individual shortcomings 

♦♦♦ multiple investigators: In addition to providing practical assistance to the Study through divisions 
of labor, the use of multiple investigators provided a diversity of viewpoints pertaining to 
methodological approaches and vantage points in examining outcomes (Dufour, 1991). The 
research teams involved in the reviewing, collecting and analyzing infonnation and data for the 
Study included the administrative and principal investigating team at TEA, the five-member 
research team fi'om the Texas A&M University~Coipus Christi, and a third-party professional 
consulting firm fi'om Austin, Texas 

♦♦♦ multiple sowces of data: The Study employed a multi-dimensional approach to data-collection 
from Agency databases, e.g., the Public Education Infonnation Management System [PEIMS] and 
the Academic Excellence Indicator System [AEIS]. The reliance upon mutiple sources ofdata incieases 
both the validity and reliability of obseived outcomes (Tnidel and Antonius, 1991) 
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QUALITATIVE APPROACHES 

The qualitative features of the multiple operations design provided depth and detail that could not 
have been realized by an exclusive experimental or quasi-experimental approach. Qualitative features 
provided enhanced dimensions of response modes to the Study questions. Researchers who utilize 
qualitative approaches may regard their task as “coming to understand and interpret how the various 
participants in a social setting construct the world around them” (Glesne & Peshkin, 1992). The 
design provided a standar dized framework for administering measures. Approaches included reviews 
of information acquired, agency data, and campus documents that relied on the following methods 
and protocols: 

❖ Teacher questionnaire and interviews 

❖ Principal questionnaires and interviews 

❖ interviews of distr ict administrators in charge of the district bilingual education program 

❖ Interviews of parents 

*1* On-site classroom visits 



TEACHER QUESTIONNAIRE AND ADMINISTRATION 

The multiple-methods used in the Study included a questionnaire that was individually administered 
to teachers, in order to collect information regarding their characteristics and professional opinions, 
with regard to the program features on respective Study campuses. The pre-survey construction 
period employed initial principles based on those advanced by Fowler, ( 1 995). These were reviewed 
and approved by multiple investigators involved with the Study. The criteria included: 

❖ The meaning of the questions and the nature of the answer mode were to be understood by all 
respondents 

❖ Respondents targeted by the questionnaire should be both willing and able to perform the tasks 
associated with the questions 

❖ The survey questions had to constitute a protocol for interaction that was consistent for both 
interviewers and respondents 

The questionnaire was completed by teacher's of record who were assigned to work with the LEP 
student population at each of the seven Study sites as identified by the campus principals. When 
analyzed, the results of the questionnaire showed that there were 101 teacher respondents. Since 
campus principals determined which teachers were to complete the questionnaire, information was 
not available to determine the percentage of teachers responding at each of the seven Study sites. 

The Program Evaluation Unit designed the teacher questionnaires and interviews in the Office for 
the Education of Special Populations at TEA [Appendix A]. The teacher questionnaires were field- 
tested in the fall of 1998 on a sample of 15% of the total number of 101 teachers, in accordance with 
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recommended criteria for sampling (Babbie, 1979; Fowler, 1995; Converse and Presser, 1989; 
Kalton, 1983). If the number of involved teachers at all campuses were to be considered the target 
population, we could calculate the standard eiTor (a measure of sampling en'or) by using the 
adapted simple formula; 

Se=^^ 

N 

where = Standard en'or rate 
P= Percentage of sample 
Q=l-P 

N= Number of cases in the sample. 

With this fonnula, as the sample size increases, the standard eiTor rate decreases. The sample of 
15% for field-testing falls well within the one standard eiTor of the parameter. 

Field-testing, or the testing of questions of the instniment, looked at how both potential respondents 
perceived meaning, response level difficulty, respondents’ interest and attention, timing, and 
respondents’ “well-being” and comfort levels (Belson, 1981; Converse and Presser, 1989; Bolton, 
& Bronkhorst, 1 996 ). This pre-testing of the questionnaire adhered to the following guiding questions 
from Czaja (1998), Fowler (1995), for respondents: 

*t* Did respondents have difficulty understanding words, terms or concepts? 

❖ Was the sentence stinicture too complex? Did respondents understand the question, the task 
required, and the answer foiTnat? 

<* Were there significant differences in time needed for respondents to answer specific questions? 

❖ To what extent did respondents have difficulty in knowing, or in providing answers? 

*t* Did respondents interpret the question as the researcher intends? 

❖ Did respondents use different response categories or choices than those offered? 

*t* Were respondents willing and able to perforai the tasks required to provide accurate and complete 
answers? 

*1* Were respondents attentive and interested in the questions? 

Other guiding questions based on the same preceding sources for field-testing were also incoi"porated 
with the other protocols to add to the pre-test validity. These included: 

Did the sections of the questionnaire and the questions provide a logical flow? 

<♦ Were instnictions coirect and easy to follow? 

❖ Was there evidence of question-order effects? 

*t* Was there evidence of ambiguity? 

<♦ What was the response rate? What were the potential problems indicated by the response rate? 
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❖ Were there any indications of problems regarding the completeness and accuracy of responses? 

❖ What was the nature of any slap patterns that may have emerged? 

❖ Were respondents’ answers within the framework of expected responses? 

*t* What types of responses were provided for questions with an “other” (please specify) category? 

❖ Was the level of variation among response categories acceptable? 

An eiTor analysis was conducted based on collected responses, which were coded for problems on a 
grid. Adjustments were made to individual wording, phrasing, and the nature of questions based on pre- 
testing responses. As a result of the field-testing, the reseaich team revised and expanded the number of 
questions on the teacher questionnaire. The questionnaires included multiple choice, yes/no, open-ended, 
and Likert-type questions. These are discussed at further length below under Fonnat of the Teacher 
Questionnaire. 

The research team relied exclusively on the responses of teachere who indicated that they were assigned 
to the bilingual education program for at least two of the three years targeted by the Study in addition to 
the 1998-99 school year when the visits were conducted. This ensured Study consistency in that eveiy 
teacher included in the Study data set had taught the LEP student population for at least three years. 



FORMAT OF THE TEACHER QUESTIONNAHIE 

Of the 65 items on the questionnaire, 16 items pertained to grade level assignments, ethnicity and 
gender. Fourteen items were used to report on the teacher characteristics across the seven Study 
sites, and 35 items were used to report on progiam features and practices. The remaining 35 items 
were divided into three categories, i.e.. Assessment Features (8), Instructional Practices (13) and 
Implementation Practices ( 14). Additionally, nine open-ended questions were administered to teachers 
using a modified Delphi-type approach. The method allowed teachers to concenti ate on probes for 
longer periods of time and encouraged their participation. The section on inteiwiew methods discusses 
the Delphi approach in gieater detail. 

Data collected through questionnaires were analyzed by the research team to obtain descriptive 
statistical outcomes. To calculate these data outcomes, a Likert-type response fonned the basis of 
an assigned score associated with each of the questions. Likert approaches utilize ordinal (rank) 
scales where the respondent indicates a degiee of intensity in agieement, disagreement, or indecision, 
as expressed by questions or statements. They are the most widely used scale types in the social 
sciences, are relatively easy to constmct, and have high reliabilities (Miller, 1997; Vogt, 1993). The 
rankings utilized on the Likeit-type scale follow: 

*1* All of the Time answers were assigned a value of 5 

**** Most of the Time answers were assigned a value of 4 

<♦ Some of the Time answers were assigned a value of 3 

<♦ Rarely answers were assigned a value of 2 

❖ Never answers were assigned a value of 1 
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The yes/no/uncertain responses, a polytomous (more than two categories) mode, were tieated in a 
different manner, since they were not on a scale basis. These responses were represented by numbers 
and percentages. Responses for the open-ended questions were summarized with succinct statements. 



INTERVIEWS 

One of the MODL approaches involved the use of interviews of teachers, administrators, and parents. 
Their use has been described by Dexter (1970) as a conversation with a puipose. A major objective 
is to obtain cunent constructions of data, reconsti\ictions, or projections of events, activities, 
organizations, and motivations among other possible information. The stmcture of interviews can 
range from open-ended unstructured fomiats, to focused interviews. Usually, stmctured or focused 
interviews becomes the mode of choice in the search for particular information. (Rubin and Rubin, 
1995; Silvemian, 1993; Strauss, 1980). It was considered that interviews could provide insights to 
several important questions and contribute to the triangulation, and ultimately, validity of data, by 
probing in several contexts, such as: 

<* What does the progiam look like on the target campuses? 

<* What were the experiences and characteristics of campus staff.^' 

*1* What were participants’ expectations as outcomes for students? 

❖ What did respondents know about their progiam? 

*> How did the progiam function from respondents’ points of view? 

*1* What concerns did respondents convey? 

*1* Which were the important features as noted by teachers, administiators, and parents? 

The strtictured inteiwiews conducted in the Study were essentially guided by protocols as part of a 
quasi-Delphi approach that was linked to the Study questionnaires. The Delphi technique is a 
pragmatic methodology employed to varying degiees in different fields. It addresses the respondent 
audience in its traditional forai as the collective “consensus of wisdom”, and involves them in 
practically all stages of questionnaire development and administration (Enzer, 1971). The modified 
approach was limited to the development of focused probes and their application within the inteiwiew 
setting for teachers. This approach allows for the development of different educational scenarios to 
be projected and evaluated by the participants. Combined with other methods, including the use of 
questionnaires and case studies, the use of the modified Delphi method contributed to a better 
understanding of the different logic of participants, and conti ibuted to the validation of outcomes 
pertaining to the educational processes on the seven campuses. 

The major methodological goal, as utilized in the Study, was to simulate a discussion process 
between the teachers from the Study sites, who influenced the daily educational processes at the 
seven target campuses. Participants in the process were given opportunities to reconsider the 
responses in conjunction with the information provided by others in order to best characterize the 
actual trend of outcomes. At two schools, the teacher interview sessions were held after school. 
Each interview consisted of reviewing the teacher questionnaire’s open-ended questions and 
clarifying any concerns that the teachers had about the Study questions. Generally, teacher interviews 
were held for about one hour to one and one-half hours. 



Interviews were held with the district leadership, campus principals, teachers, and a significant 
number of the parents of the LEP students at each of the Study sites. School administi'ator interviews, 
using the Study interview protocol; were held for about one hour, usually upon entering the school. 
Most teacher interviews were held during the school day, with the campus principals an'anging for 
teacher aides, other teachers not on duty, or volunteer parents to oversee the classrooms, while the 
teachers were interviewed by the research team. 

Parent interview sessions were conducted at all of the seven school sites. Parents were selected 
from a list provided by the principal. The list included members of the site-based campus team and 
the Parent Teacher Association/Organization. A parent interview protocol was administered in 
English and in Spanish as appropriate; however, most of the interviews were conducted in Spanish. 
Parent interview sessions lasted for one hour to one and one- half hours. 



CASE STUDIES 

Case studies, as a qualitative approach of the design, provided reflexive reporting that depicted 
mutually shaping influences at work on Study campuses. Utilized primarily in descriptive, 
explanatoiy and exploratory research, case studies may be singular, or multiple in nature. Their 
methods do not automatically imply any specific form of data collection which may combine 
qualitative and quantitative approaches (Chen, 1990; Yin, 1993). Case studies analyze each 
phenomenon in its natural context, i.e., that which is actually occulting, including the point of view 
of the participants. In case studies, the researchers spend an extended period of time on-site with 
the research participants. A substantial amount of data is gathered fi'om a wide variety of sources to 
present a description of the phenomenon or experience fi'om the perspective of the participants. 

Through the approaches inherent in the MODE design, it was possible to profile the characteristics 
for each campus over a three-year period, while profiling assessment, instructional and 
implementation practices actually reported, analyzed and observed. The campus profiles were based 
on teacher questionnaire responses, administi'ator, teacher, and parent interviews, and on-site classroom 
obseivations for 1998-99. The case studies for each campus, individually and collectively, gi'eatly enhanced 
the total picture of what was actually occumng with limited English proficient students. The success of 
eveiy school profiled in the Study is documented with inforaiation presented in each case study fi'om the 
perspective of the Study participants, and supplemented by agency data available. 



ON-SITE VISITS 

Classroom visits were conducted in almost all classrooms in all of the seven Study sites. The 
classroom obseiwation visits consisted of one research team member visiting in a classroom for 
approximately 20-30 minutes. The member scripted observations or wrote anecdotes of activities 
in the classroom that impacted the learning of language minority students. Assigned teachers of 
record for the LEP population, as well as non-assigned teachers for the LEP population were visited 
in their classrooms. Obseiwation focused on the instiuction of the cognitive, affective and linguistic 
needs of LEP students. Pursuant to conducting a visit each day, members of the research team 
convened to share and validate significant features and practices observed that could impact the 
success of the LEP students. By using this approach, the research team was able to discern 
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commonalities of obsei'ved practices. School visits, classroom visits and all interview sessions 
were very pleasant and well received by all the participants. Admin isfrators (district and campus), 
teachers and parents were willing to share their responses candidly and openly. 

The campus visits were conducted during March, April and May 1999 by the research team. The 
on-site visits to the Bowie and Clover campuses in the Pharr-San Juan- Alamo Independent School 
District were observed by the Agency’s Principal Investigator to ensure that the visits adhered to 
the study procedures. Intei'views of campus principals were conducted during the on-site visits by 
members of the research team at each study site. A calendar for the school visits was developed in 
consultation with the campus principals as follows: 



March 11-12 


Visit Castaneda Elementary 


Brownsville ISD 


Brownsville, Texas 


April 


14-15 


Visit Campestre Elementary 


SocoiTO ISD 


El Paso, Texas 


April 


29-30 


Visit Scott Elementary 


Roma ISD 


Roma, Texas 


April 


29-30 


Visit Kelly Elementary 


Hidalgo ISD 


Hidalgo, Texas 


May 


04-05 


Visit Clover Elementary 


PhaiT-San Juan-Alamo ISD 


San Juan, Texas 


May 


04-05 


Visit Bowie Elementary 


Phan-San Juan-Alamo ISD 


Alamo, Texas 


May 


06-07 


Visit La Encantada Elementary 


San Benito CISD 


San Benito, Texas 



At least two members of the research team were present at each school during the two-day visit to 
the Study sites, with the exceptions of Clover Elementary and Bowie Elementary. At these two 
sites, the research team was expanded to five members to conduct the visits and collect all data in 
one day at each location as requested by the campus principals. The one-day visits were considered 
to be less disruptive in these two schools that were making preparations to close a school year. 



QUANTITATIVE APPROACHES 

Quantitative approaches involve the numerical representation and analyses of data. The quantification 
of data can be utilized in both einpirical and naturalistic designs (Brannen, 1992; Greene and 
Caracelli, 1997; Patton, 1990) Quantitative measurement relies on instruments, statistical approaches, 
and presentations which provide standard frameworks that can both limit and assign data to 
predetennined categories of outcomes and analyses. Simply defined, quantitative methods yield 
greater breadth of data and more accurate descriptions for questions such as “what” and “how 
much,” while qualitative approaches lend greater depth of comprehension and accuracy for the 
“why” and “how” questions. The following sections describe the quantitative approaches to data 
employed by the Study. Further detail regarding specific statistical foiTnulas are found in Appendix 
B of this document. 



DESCRIPTIVE STATISTICAL APPROACHES 

Descriptive statistics refer to properties of distributions, such as location, dispersion and shape. 
Descriptive statistics involve the collecting, categorizing and presenting of data. They are used to 
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describe the basic features of the data in a study that provide simple summaries about the sample 
and the measures. Together with simple giaphics analyses, they form the basis of viitually every 
descriptive quantitative analysis of data. 

Descriptive statistics are typically distinguished fiom inferential statistics. With descriptive statistics, 
one is simply describing what is, or what the data shows. With inferential statistics, there is an 
attempt to reach conclusions that extend beyond the immediate data; whereas descriptive statistics 
simply describe the outcomes through the use of numbers. Descriptive Statistics are used to present 
quantitative descriptions in a manageable fonri. Descriptive statistics attempts to simplify large 
amounts of data fiom multiple measures. Each descriptive statistic reduces data into simpler numeric 
summaries. The single number describes a large number of discrete events; for example, a student’s 
percentage score in passing TAAS. This single number describes the general performance of a 
student across a potentially wide-range of perfoimance experiences. Where comparative statistics 
can describe the relationships between variables, e.g., how changes in one variable relate to changes 
in another, descriptive statistics depict the values taken on by one variable at a time. In addition to 
numerical counts and percentages, descriptive approaches for the Study also employed data displays 
and measures of cential tendency as described below. In the discussion of interjudge reliability of 
the teacher questionnaire used in the Study, it is indicated that additional analyses, such as non- 
parametric foiTnulations were perfoiTned on a limited post-hoc basis. These foiTnulas may be 
discussed in a future study, since they had not been integrated within the scope of the initial Study. 



DATA DISPLAYS 

Various types of data displays were employed by the Study. These displays represented different 
views of data in numerical temis. They included tables, cross- tabulations, bar, and line graphs. 
Although line graphs were not utilized in the Study, they were subsequently developed and used by 
the Program Evaluation Unit and members of the university research team in numerous {over 20} 
presentations on the Study design and findings at statewide and national workshops/conferences. 

Tables [basic] were utilized to display data through the use of columns and/ or rows. These 
descriptive displays were both numeric and in nanative fonri 

❖ Cross-Tabulation is a combination of two (or more) frequency tables ananged in such a way 
that each cell in the resulting table represents a unique combination of specific values. Usually, 
the cells also contain percentages. Cross-tabulation allows us to examine fiequencies of 
obseivations that belong to specific categories on more than one variable. By examining these 
frequencies, we can identify relations between cross-tabulated variables. The cross-tabulation 
tables used in the initial analyses of the Study to quantify teacher responses depicted the number 
of cases having specific combination of values 

❖ Bar Graphs in the Study illustrate student perfoiTnance outcomes on the TAAS, either between 
the target and external campus cohorts, or the target and peer campus comparison gioup. The 
height (horizontal bar graph), or length (vertical bar graph) of a bar describes the fiequency of 
the obseivations in that gi oup of students (each gi oup has its own bar). Samples of bar graphs 
developed for Study presentations are found in Appendix C of this document 
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Line Graphs are a visual representation of several sets of related data. The name is given to a 
graph because of the individual points which are joined by a line or lines. As mentioned above, 
line giaphs were not used in the Study, but were incorporated into presentations made at numerous 
workshops/conferences. [Appendix C] 



Measures of Central Tendency and Variability 

Descriptive statistical approaches to data in the Study, particulaiiy for outcomes from the Teacher 
Questionnaire, utilized both measures of centi al tendency and variability. The narrative that follows 
describes the nature of these measures and their use. 

Measures of central tendency are statistical summaries of data. They are intended to reflect a single 
number, which represent other numbers, and indicate location. Most often used measures of central 
tendency are the mean, the mode, and the median as described below. 

❖ Mean is a particularly informative indicator of cential tendency for a variable if it is reported 
along with its confidence intervals. Usually, there is interest in statistics (such as the mean) 
from the sample only to the extent to which they are inforaiative about the population. The 
larger the sample size, the more reliable its mean. The larger the variation of data values, the 
less reliable the mean. 

❖ Mode of a sample is the value which occurs most frequently in the distribution. It is possible for 
outcomes to have two, three, or more values, which will occur with the same frequency. In 
these instances, the outcomes take the form of being bimodal, ti imodal, etc. 

❖ Median of a distiibution is the value for which one-half (50%) of the observations (when ranked) 
will lie above that value, and one-half will lie below that value. When the number of values in 
the distribution is even, the median is computed as the average of the two middle values. Because 
the median is less sensitive to extreme scores than the mean, this makes it a better measure than 
the mean for highly skewed distributions. If the disti ibution of numbers is symmetrical, such as 
that ofthe bell curve, the mean and median will be equal. The calculation of the median however, 
using & grouped frequency distribution such as the Likert-Xygt 5-point rating scale, constrains 
responses to a small set of discrete values when the underlying attiibute being measured is 
really a continuous scale. 

Measures of Variability depict the spread, or dispersion, of outcomes within a giouping. The measures 
of variability more foraially show the deviation of the distribution from a central tendency measure 
such as the mean. Standard deviation and variance are the two most utilized measures of variation. 
Within the Study, the analyses of data employed the standard deviation. 

Standard Deviation is a commonly used measure of variation and dispereion. This statistic tells us 
how tightly outcomes in a data set are clustered around the mean. The more the scores differ from 
the mean, the larger the standard deviation. The standard deviation of a population of values is 
utilized to calculate all of the data in a population. Knowing the mean and standard deviation of a 
sample, or a population, can provide location for most of the data values. 
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BENEFITS 



The MODL design provided multiple benefits of multiple methods [Appendix D] and data gathering 
approaches that allowed investigative and data depiction features, which were descriptive, 
exploratoiy, explanatoiy, and confinnatoiy as described below: 

❖ Descriptive investigations depict phenomena in contrast with quasi-experimental studies where 
environments are conti'olled and subjects receive various treatments. Using words or numbers, 
descriptive research depicts conditions as they exist in the particular setting(s). The approaches 
in the MODL design essentially illustrate settings, activities, participants and extent of 
participation, and are designed to describe the characteristics of the target population under 
investigation. Generally, descriptive research designs may be classified as either qualitative 
and/or quantitative 

❖ Exploratoiy investigations usually represent the initial steps of a research effort (Tashakkori 
and Tedlie, 1998). Essentially, researchers who utilize exploratoiy approaches seek to understand 
and describe relationships between identified outcomes or phenomena. As the terai would suggest, 
exploratory investigations occur to clarify the scope of the research and help focus the research 
questions. One of the major benefits of these approaches is the viability to exti'apolate ideas and 
insights. This, in turn, can lead to a familiarization with the problems or concepts under study, 
and the generation of research questions that are relied upon to guide a research project 

K* Explanatoiy investigations ti'aditionally seek to convey an understanding of both anticipated 
and unanticipated patterns and relationships between observed outcomes. In empirical models, 
the rate of an event occuiring varies according to a set of explanatory variables (Tashakkori and 
Tedlie, 1998). Although the Study did not employ the rigorous classical empirical assumptions 
of explanatoiy investigations (e.g., delineated independent variables), the MODL paradigm 
incorporated features to help clarify outcomes, rather than investigating causal links. This was 
accomplished in descriptive foiTnats through the quantitative visual display of data (Miles and 
Hubennan, 1988). These approaches permitted the depiction of experiences fi'om the perspectives 
of diverse stakeholders and observed points of convergence and divergence in responses. The 
flexibility of the MODL design, however, allowed for the use of more than one case Study to 
assess explanatory features. This aligns with explanatory, or descriptive studies, that generally 
require multi-site case studies to search for patterns or themes, and histories and ethnographies 
to understand the meaning of the experiences of the participants in the culture. Both exploratoiy 
and explanatoiy features of the MODL design helped identify the concepts and the bases for 
measurement, and were very appropriate for early stages of the research. Explanatoiy approaches 
in the MODL design facilitated triangulation. or the use of multiple methods (both qualitative 
and quantitative) to cross-check each other, enhancing confidence in findings (Keiiinger, 1986; 
Dubin, 1978) 
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❖ Confirmatory investigations traditionally employ factor analyses, within the frameworks of 
empirical designs (Long, 1983). It was not the intention of the Study to incoiporate stinct 
inteipretations of confirmatory analyses since it did not seek to prove hypotheses, empirical 
causality, or to employ predictive models. Nevertheless, the MODL design allowed for 
triangulations and various cross-validations resulting in confimiatory features particularly in 
qualitative approaches. These included surveys, obsei-vations and interviews. Site visits and 
classroom observations in schools maximized consistency, expanded on the survey outcomes, 
and also allowed the construction of case studies of individual schools. Results of the Study 
were additionally confirmed by the presence of comparison gioups of students and associated 
testing outcomes 

RESEARCH QUESTIONS 

An initial search and analyses of the literature resulted in listings of preliminary questions. These 
questions in turn formed a logical chain in identifying “relevant knowledge.” A more comprehensive 
review of ovei' two hundred sources resulted in a more nan ow definition, and the crafting of inherently 
interdisciplinary research questions pertaining to successful practices in educational settings. A 
review by multiple investigators further led to further refinement of the guiding questions for the 
Study, and resulted in a “consensus of wisdom” in support of the finalized form of the questions. 
[Appendix E] Important considerations ensured that the questions were concise, clear, focused, 
testable, and intenelated to one another. The research questions adopted for the Study were: 

♦J* What are the LEP, former LEP and non-LEP students’ academic performance as measured by 
state (Grades 3-5) assessments? 

❖ What are the district leadership practices that facilitate academic and linguistic growth/success 
for language minority students? 

❖ What are the campus leadei-ship practices that facilitate academic and linguistic giowth/success 
for language minority students? 

What are the characteristics of the teaching staff that facilitate academic and linguistic giowth/ 
success for language minority students? 

❖ What are the effective teaching practices that facilitate academic and linguistic giowth/success 
for language minority students? 

*** What are the characteristics of parents and parental involvement on the seven campuses? 

*** What are the characteristics of progiam(s) sei'ving language minority students? 

SOURCES OF DATA 

The multiple sources of data were elicited fiom five major sources. These included infomiation 
from the Academic Excellence Indicator System (AEIS) reports from each of the four years 
referenced, and data fiom the Public Education Infomiation Management System (PEIMS) data 
base. These data addressed campus and individual student perfonuance on the TAAS for each of 
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the two Study cohorts, as well as the TEA peer campus group comparisons. In addition, data were 
obtained fi'om interviews, survey questionnaires, and on-site visits to discern student and campus 
outcomes. Illustration 1 depicts the multiple sources of data features for the design: 



The AEIS reports are produced annually and contain detailed information, e.g., by subject, by 
gi'ade, by ethnicity and by program type, regarding student and campus perfonnance on individual 
campuses (6,000+), and by districts (1,000+) in the state. These comprehensive reports provide 
information such as student perfonnance, demogi'aphic background on students, teacher, program 
and financial information. Student and campus performance, in addition to dropout rates and 
attendance, are used in the Texas Accountability System to assign accreditation ratings to campuses 
and school districts. AEIS reports were downloaded fi'om the TEA web site for each campus in the 
Study. These reports were used to generate descriptions of the campuses within the Study as well as 
to validate student and campus perfonnance. 

Infoimation regarding student perfonnance included the percentage of students passing and mastering 
all objectives on each TAAS subtest. This enabled analyses by objective and by grade level for 
different student groups. It also allowed for the examination of perfonnance by grade level over 
several years, as well as progress by students as a group (quasi-cohort). These data were used to 
confiim findings and to expand upon the level of detail contributing to the explanatoiy characteristics 
of the design. The outcomes reported in the Study relied exclusively on perfonnance data of students 
who had been promoted from gi'ade to grade since initial enrollment in Kindergarten as LEP, and as 
bilingual education program participants. 



ILLUSTRATION 1 

MULTIPLE SOURCES OF DATA 




Principals were also asked to submit data available on oral language proficiency tests in English 
and Spanish for LEP students in Grades PreK-5, as the students progressed through the bilingual 
education program. This request for oral proficiency data was made in order to address one of the 
research questions of the Study, pertaining to the development of linguistic proficiency. In order for 
these data to be reliable, both pre and post test scores on each LEP student needed to be recorded by 
the LPACs during a given school year. Since many of the LEP students were administered an oral 
language proficiency test upon entering, progressing within, and/or exiting the bilingual education 
program, this process for data collection did not yield consistent and reliable campus data across 
the seven Study sites, except in one instance. The exception applied to the only campus with a 
Grade K-3 configuration and the results of the oral language proficiency of the LEP students at that 
campus [Scott Elementary] are presented in the final Study document. 

As part of the methodology. Study questions were matched on a grid with available sources of data. 
The documentation of processes enhanced the conduct of methods and contributed to the validity 
of the design by ensuring consistent applications. 



The MODL approaches included both student cohorts and comparison groups on TAAS performance 
in examining student outcomes for the years in which data were available. Study [student] cohorts 
were fonned from both target and external campuses based on enrollment in Kindergarten, 
classifications as LEP, and served in a bilingual education progr am for either of two years, i.e., 
1994 Cohort and 1995 Cohort. Both student cohorts were tracked using a vertical progression 
model, until the data showed that students had transitioned fi'om LEP to former LEP status. During 
the 97, 98 and 99 school years, TAAS data became available to assess performance of all students 
that continued in LEP status, had transitioned to fomier LEP, or were originally identified in the 
Study cohorts as never LEP. It is for these three years that Study or target campuses could be 
compared to the external campuses, whereas target campuses could only be compared with the peer 
campus group during the 1998-99 school year. 



STUDENT COHORTS/COMPARISON GROUPS 



ILLUSTRATION 2 

STUDENT COHORTS/COMPARISON GROUPS 



Target and 
External Campuses 



Target and 
External Campuses 



Target and 
TEA Peer Group 




Since the MODL design did not employ strict hypothetical-deductive approaches, it was not concerned 
with measuring inteiwentions, or in employing theoretical tieatment and conUol gioups. In profiling 








student performance, the MODL design took advantage of two oppoitunities to utilize compaiison, and 
not contr ol gr oups. One of these opportunities presented itself as an initial limitation of the design. As 
within any cohort study, students may be lost over a significant period of time. This loss could have been 
a methodological concern for the Study, especially at the individual campus level. By utilizing the PEIMS 
data system at TEA, this was not a concenr for the Study, since students could be tr'acked regardless of 
their location within a public school in Texas, either as LEP or former LEP. There were no problems in 
identifying the original cohorts of students through the assignment of student identification number's. In 
this instance, students who left the target schools, but remained in an “external” campus within the state, 
actually provided str ength to the Study. These students formed a general comparison group in contrast 
with those that remained in the or'iginal target schools participating in the Study. 

Once again, it is not infened, nor was it ever an intent of this Study to contend, that these identified 
students formed a tr'ue compar-ison, or control gr oup, along the lines of theoretical experimental, or 
quasi -experimental designs. Any consideration of an empirically defined control gr oup was further 
dispelled with the identification of most of the students who had tr ansfen ed to another campus as being 
enrolled in a bilingual progr am. 

A second opportunity presented itself with the formation of a TEA compar-ison campus group associated 
with each tar get school cohort. These compar-isons between each Study cohort and the TEA peer campus 
compar-ison group added further r-igor to the descr-iptive nature of the design. The collective TEA 
compar-ison campus gr oups had been computed by TEA as part of the Academic Excellence Indicator 
System (AEIS). They were der-ived by identifying a unique comparison group of 40 campuses, on the 
basis of six demographic characteristics defined in statute, and found to be statistically related to 
performance. The character-istics are: 

<♦ Per cent of Afi-ican Am er-ican students enr olled for 1 998-99 
<♦ Percent of Hispanic students enrolled for 1998-99 
<♦ Percent ofWhite students enrolled for 1998-99 
<♦ Per cent of economically disadvantaged students enrolled for 1998-99 
*1* Percent of limited English proficient (LEP) students enrolled for 1 998-99; and 
Percent of mobile students as determined fi om 1997-98 cumulative attendance 

With the incorpor-ation of target, exter-nal, and peer campus compar-ison gr oups, the MODE design 
allowed for four (4) different observations over time; 

*1* Performance Of the overall student population on the Study campuses as a composite gr oup 
❖ Performance of the Study campuses compared to the TEA compar-ison campus gr oup 
*1* Performance of the Study campuses examined individually 

*1* Performance of the cohorl gr oup on the Study campuses compared to students in the cohorl who 
tr-ansfened to other campuses on an intra or interdistr-ict basis 
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OTHER FEATURES 

Although the design of the Study had adopted some characteristics of positivistic quasi-experimental 
approaches, it conformed more in its direction to practices associated with the naturalistic inquiry 
paradigm. The uniqueness of the MODL design allowed for a comprehensive strategy in 
understanding and describing the nature of the educational and parental participation processes at 
the seven participating schools. Since the Study was more descriptive in nature, and did not pursue 
the testing of hypotheses, or the investigation of causality, the stinngent controls associated with 
quasi-expenmental designs are not evident. 

The nature of the Study did not allow for the strict incoiporation of quasi-expeiimental methods. 
For example, the post-hoc approaches to selected student data and availability of Study populations 
mitigated against random selection and assignment. The Study could not realistically consider these 
cohorts as theoretical classical conti ol groups. Even if such theoretical conti ol groups, governed by 
canons of experimental research were possible in reality, their fomiation [identification] could 
have raised challenges under federal and state statutes which prevent the deprivation of an appropriate 
program, e.g., educational tieatment required for students with special needs. Instead, the Study 
employed contiast gi oups, i.e., external campus groups and peer campus comparison groups, which 
were composed of students that were matched with Study cohorts on target campuses, based on 
similar characteristics. 

The next subsections address issues of validity and reliability pertaining to the Study. This manual 
makes no attempt to address the validities and reliabilities of the TAAS . Information pertaining to 
the nature and structure of the TAAS, including its backgi ound, development, reliability, validity, 
and other technical features are available in a series of Technical Digests for respective academic 
years that TAAS was administered. These digests are not included in this manual, but are available 
by contacting the Student Assessment Division at the Texas Education Agency. 



VALIDITIES 

Technically, validity has been usually defined as the extent an instrument is actually designed to measure. 
As Cronbach (1971; also Chen and Rossi, 1987) had noted decades earlier, validity applies to the 
inteipretation of data of outcomes fi om specified procedures. The focus on the validity of a design, an 
obseiwation, or an instiniment, reflects whether or not outcomes are true, identified and labeled 
appropriately. Several types of validities relate to the design of the Study. These include instrumental, 
internal and theoretical validities. 

Instrumental validity, also called criterion and pragmatic validity, traditionally addresses whether 
or not generated obseiwations from one instrument match those from an alternative measure 
(Kirk and Miller, 1986; Nunally, 1978;). Within the MODL fiamework, the definition refere to 
alternate procedures, rather than to different foiTns of the same measure. An example is the use 
of valid stinictured interviews referenced to the same criterion of a separately administered 
questionnaire. The employment of pattern identification, tracking, and matching of data outcomes 
added to the validity of instmments utilized by the Study. (Ragin, 1990;Trochim, 1985; Tinidel 
andAntonius, 1991). 
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<* Internal validity tiaditionally has been applied to causal studies and causal relationships. Within 
the MODL framework, it applies to the explanatory features of the paradigm by describing a 
chain of events that lead to obseiwed conditions, rather than from spurious associations. (Kidder 
and Judd, 1 986). For example, in the multiple case study methodology employed by the design, 
internal validity was discerned through: A) pattern matching, which revealed similarities in 
outcomes, b) explanation bnilding, where the investigation focused on how and why the outcomes 
occuned across cases that eliminated rival explanations (Yin, 1994; Yin and Moore, 1984; 
Glaser and Strauss, 1967). This last condition anticipates that the outcomes were not the result 
of an earlier intervention. 

❖ Theoretical validity occurs when substantial evidence arises to support the conespondence of 
outcomes to the theoretical paradigm. It also underlies and supports the concept of instiaimental 
validity. (Kirk and Miller, 1986). Although the Study did not pursue the testing of hypotheses, 
there was evidence of theoretical or construct validity. There was substantial evidence in the 
Study, for example, to associate student outcomes and successful educational practices at the 
seven campuses with the Effective School Conelates (1998). 

The three tests of validity of the Study are summarized below. Included are some of the Study 
strategies, based on a model developed by Yin (1994), that conti ibuted to each particular validity of 
the Study. Each of the Study stiategies contributed to the robustness of the design, and were in line 
with similar general concepts adopted from two sources: 1) the U. S. General Accounting Office 
(1990) of credibility, ti'ustworthiness, data dependability, and confirmability and 2) Lincoln and 
Guba, (1985) credibility, transferability, dependability, and confinnability. 



TABLE I 

STUDY STRATEGIES FOR THREE TESTS OF VALIDITY 



Validity 


Study Strategies 


Instrumental Validity 


Standardized methods and protocols 
Multiple sources of data outcomes 
Matching data from alternate processes 


Internal Validity 


Multiple sources of data outcomes 
Pattern investigation 

Logical chain of explanation for data outcomes 


Theoretical Validity 


Multiple sources of data outcomes 
Establishing chain of conesponding patterns 
Matching pattemed outcomes with paradigm 
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RELIABILITIES 



The classic definition of reliability refers to the extent a measure yields the same results after 
repeated trials (Carmines and Zeller, 1979). Yin (1994) takes the definition further by noting that 
reliability addresses if another investigator, following the same procedures with comparable 
populations, would yield similar outcomes. Exclusive experimental or quasi-experimental designs 
have a propensity to pursue research intended to be confirmatory as proof of generalizability. When 
all goes well, the outcomes support reliability of an initial study. At times, external validity and 
reliability have been utilized in the same thought processes by definition of generaUzability. Using 
Yin’s definition, the independent replication of the event through the same procedures distinguishes 
it from external validity. In contiast, the multiple operations model allows for several views of both 
external validity and associated reliability. If one views each of the seven schools as individual 
entities, subject to the application of multiple investigatore, the outcomes in student perfoiTnance 
were similar from campus to campus, as were the successful practices associated with the 
perfonnance. In the instance of the Study, the definition would apply to the stability of the design 
fi'om one use to the next. From the perspective of the Study, outcomes seemed to support aspects of 
both rf;oc///omc and iy/7c///omc reliabilities: 

❖ Diachronic reliability refers to the stability of observed outcomes over a period of time (Kirk 
and Miller, 1 986). The concept calls for outcomes that need not be exactly identical, but similar 
over a period of time. Theoretically, in its sti'ictest interpretation, the concept occurs when 
features and entities remain basically unchanged, posing problems for educational research. 
Within the context of the Study, the definition is not utilized in its sti'ict isomorphic configuration 
of outcomes. Outcomes from the Study lend themselves to aspects of the concept over an 
obsei-ved period of time. Study campuses exhibited outcomes of student success over a number 
of years. Associated with this success over the same period were the applied successful school 
practices. The extent of their application was not an objective of the Study. Nevertheless, data 
outcomes revealed the consistent presence of these practices over the same time period for all 
seven campuses. In addition, some fonn of external validation of these efforts occuired over 
these years as these campuses displayed similai' patterns of receiving either “Recognized” or 
“Exemplary” status fi'om the TEA. 

❖ Synchronic reliability refers to the similarity of outcomes within the same period of time. As 
with diachronic reliability, these outcomes need not be identical, but should reflect consistency 
(Bickman, 1986). Synchronic validity mimics internal validity where it can be detemiined 
through multiple sources of similai' outcomes fi'om different measures. However, these measures 
are consistently applied at different sites. In the instance of the Study, standardized approaches 
were applicable to all Study sites. Similar patterns of student success and effective educational 
practices were also evident within a framework of the same delineated periods of time across 
the seven campuses. 

Both diachronic and synchronic reliabilities are reinforced when interjudge reliability is attained. 
In terterjudge reliability, or interobseixer reliability, refers to the degree of consistency by observers 
in their ratings (De Vellis, 1991). This is usually conducted through deteiTnining the degree of 
agreement between respondents. From a qualitative perspective, the Study assessed the degree of 
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agi'eement of intei-view responses and outcomes from administered questionnaires. From a data 
analysis perspective, consistently small standard deviations indicated the degree of consistent 
agreement in analyzed responses. 

The two tests of reliability for the Study ai'e summarized below. As with the tests of validity, included 
are some of the Study sti'ategies, based on a model developed by Yin (1994) that conti'ibuted to each 
particular reliability of the Study. Once again, each of the Study strategies contributed to the 
robustness of the design. These sti'ategies also conformed to the general concepts previously noted. 



TABLE n 

STUDY STRATEGIES FOR TWO TESTS OF RELIABILITY 



Reliabilty 


Study Sti'ategies 


Diachronic 

Reliability 


Applied standardized methods and protocols to individual sites 


Multiple sources of data outcomes for a number of yeare 


Matching consistent data outcomes from individual sites 


Confii'ming 


Synchronic 

Reliability 


Multiple sources of data outcomes within fixed period of time 


Inter-site pattern investigations 


Establishing chain of coiresponding patterns 


Confii'ming inferential consistency of data patterns for fixed period of time 



Further discussion of particular validities and reliabilities as they apply to data gathering 
approaches, such as the Teacher Questionnaire, Intei-views, and Case Studies are provided in 
Appendix F of this document. 
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III. SUMMARY OF STUDY FINDINGS 



T his section is intended to provide a listing of the findings that can be, or have been, documented 
fiom data and infomiation contained in The Texas Successful Schools Study: Quality 
Education for Limited English Proficient Students final report and post-hoc analyses 
conducted by the Principal Investigator for the Study. As documentation to support the findings, 
several tables are also included in this section, immediately following the list of findings. Although 
the Study itself did not have a dedicated research effort to test hypotheses, causality, or seek to 
explain relationships beyond employing descriptive methods, a further “study of the Study” provided 
evidence to support the findings as listed below: 

1 . Hispanic enrollment in Texas increased as a percent of total enrollment from 3 1 percent in 
grade 12 to 42 percent in Kindergarten, indicating more Hispanic students are entering at 
the early giades. Conversely, the proportion of white students steadily decreased from Grade 
12 to Kindergarten 

2. Minority students represented 89 percent of all new student growth in Texas public schools 
over the four-year period reported, while non-minority students represented 1 1 percent of 
the new enrollment [Table III] 

3. An analysis for a six-year period between 1991-92 and 1997-98 indicates that Texas public 
schools experienced an increase of 44 percent in the LEP population [Table IV] 

4. Approximately 85 percent of all new LEP enrollment was evident in six of the twenty 
service center areas; all in urban or statistical metropolitan areas [Table IV] 

5. Seventy-seven percent (77%) of all new LEP students were enrolled at the elementary 
grade levels in 1997-98. [Table V] 

6. The categories utilized for LEP student identification in six of the seven Study sites were 
“Beginner,” “Intermediate” and “Advanced” 

7. The program offering for LEP students was enhanced by coordinating different funding 
categories that included the Foundation adjusted basic allotment; Bilingual allotment (10% 
of the adjusted basic allotment). State Compensatory Allotment (20% of the adjusted basic 
allotment). Title I Regular/Migrant, Title VII Bilingual Education, and the Emergency 
Immigrant Education Program 

8. Based on questionnaire responses, over 9 1 percent of teachers assigned to the LEP population 
at the seven study sites were Hispanic 

9. Based on the numbers and percentages of actual questionnaire responses received, 85 percent 
to 100 percent of the teachers were trained in bilingual methods; trained in language 
assessment; knowledgeable of the benefits of second language learning, and confident in 
their training to address the needs of LEP students [Table VI] 



1 0. The results of two of the assessment features surveyed for the Study show that there is a tendency 
to assess English proficiency more often than Spanish proficiency. According to the Rules of 
the Commissioner, e.g., 19 TAC Chapter 89 SubChapter BB, assessment of LEP students in 
English is required to reclassify students who meet the required exit criteria to Non-LEP status. 
This procedural requirement invariably results in a greater effort to assess the English language 
than there is to assess the Spanish language 

1 1 . The results of the questionnaire on instructional practices indicate that there appear to be more 
teachei'S that have a system to provide English instiuction than those who have a system to 
provide Spanish instruction. In the absence of an item analysis of the teacher questionnaire by 
teacher and grade levels, it is not possible to document if the responses were from teachers in 
the upper grades, e.g., 3’^'*, 4"’, and S"'. It is at these gi-ade levels that LEP students, who have 
been in the bilingual education program for four and five years, begin their transition to the 
second language (English). At this ti'ansition point, the instiuctional focus is usually in the 
English language 

12. While it may appear that the number of new teachers hired, in comparison to the number of new 
students enrolled in Texas public schools, is adequate to address the problem of teacher shortages, 
such is not the case. When attention is focused on where the new students are enrolling, and 
who the new students are, the total state picture of teacher increases does not align with the 
changing student demogi'aphy [Tables VII-VIII] 

13. All of the principals of the seven Study sites were Hispanic women with mid-management, 
elementai-y and bilingual education certification. Six of the seven administrators had over 20 
years of total experience, and one had between 15 to 19 years experience. All of the principals 
have had administrative experience with bilingual education programs as part of the 
administi'ative responsibilities, and all had more than 5 years experience teaching in a bilingual 
classroom [Table IX] 

14. Texas is experiencing a teacher shortage that is a serious and growing problem. The certification 
programs are not producing enough teachers, both to staff increased enrollment and replace 
teachers who are promoted, or leave the profession 

15. Bilingual education was provided to the LEP students as integral parts of the regular school 
program in all seven Study sites 

16. The Study sites implemented the appropriate program by focusing on the cognitive, linguistic 
and academic domains to ensure that LEP students become competent in the comprehension, 
speaking, reading and composition of the English language 

17. The academic performance of 3'^'* Grade LEP students in the seven Study campuses 
significantly exceeded the performance of 3'^'^ Grade LEP students in the cohort comparison 
group in external campuses 

18. The academic perfonnance of former LEP students in 5"’ Grade in the seven Study campuses 
exceeded the perfonnance of 5"’ Grade foimer LEP students in the cohort comparison group in 
external campuses 
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19. The most significant difference in 5''’ Grade academic perfonnance between fonner LEP students 
in the seven Study campuses and former LEP students in the cohort of external campuses, was noted 
when students had been in the bilingual education progi'am for 5 and 6 yeai-s. This difference in 
academic performance in favor of the taiget LEP population is evidence of the sustained effects of 
the bilingual education progium provided in all of the seven Study schools. [Note: Since the LEP 
cohorts for the Study were based on Kindeigarten enrollment, LEP classification and bilingual 
education program participation in PreKindeigaiten were not included in the number of years reflected 
in the performance gi'aphics in Appendix C. 

20. In the Late-Exit model, the exiting of LEP students was more evident in Grade 4 and Grade 5, 
after students had been in the bilingual education program for 6 and 7 years 

2 1 . Each of the seven Study sites were above the state average on teacher to pupil ratio in 1 997-98; 
six of the seven sites were also above the state average on teacher to pupil ratio in 1 996-97, and 
five of the seven sites were above the state average on teacher to pupil ratio in 1995-96 

22. Three of the Study sites were above the state average on instmctional per pupil expenditures for 
each of the three years targeted by the study; two of the Study sites were below the state average on 
instmctional per pupil expenditures for each of the three yeai's, and the remaining two campuses 
were below the state average in at least one of the three yeai's targeted by the Study 

23. Although all of the seven Study sites reported a teach er-to-pupil ratio above the state average 
in 1997-98, six of the Study sites were rated as “exemplary,” and one was rated “recognized” 

24. Transition criteria from Spanish to English was accomplished by using both languages as 
mediums of sequential instmction in the affective, cognitive and linguistic domains 

25. Transition to all-English language instruction was not evident for LEP students in the 
“Beginner” level 

26. LEP students were classified as English proficient, e.g., Non-LEP when scoring at or above the 
40''’ percentile on the English reading and English language arts sections of a noi'm-referenced 
assessment insti-ument, or at the end of the school year in which a student would be able to 
participate equally in a regular all-English instmctional program as evidenced by mastei^y of 
the criterion-referenced assessment insti-ument (TAAS) requirement as found in the Texas 
Education Code 

27. Eighty-seven percent (87%) of all teachers reported educational experience ranging from 
5 to 20 years; 83 percent of all teachers reported bilingual education experience ranging 
from 5 to 20 years. 

28. Eighty-nine (98%) of the 91 teachers that responded indicated they assessed the levels of 
both primary language (Spanish) and English to ensure an appropriate instmctional focus. 
Of the 89 teachers, 82 (93%) indicated they did the assessment on an on-going basis during 
the school year 

29. Eighty-seven (87%) to 100 percent of the teachers across the seven Study sites indicated they 
modified instruction and placement of LEP students upon receiving new infomation fi'om on- 
going assessments 
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30. When responding to allowing LEP students to express themselves in Spanish versus English, 
the responses indicate almost equal results with 86 (95%) responding “Yes” to Spanish, and 87 
(96%) responding “Yes” to English. These results document that both languages were given 
equitable prestige 

31. Principal and disti-ict leadei-ship support for LEP students received almost equal responses, 
with principal support having 84 (94%) “Yes” responses, and district leadership support having 
82 (93%) “Yes” responses 

32. Eighty-five percent (85%) of the teachers surveyed indicated that parental involvement helped 
LEP students advance in academic development, and 75% of the same teachers rated parental 
involvement as helping LEP students advance in their language development 

33. The cumulative total for ti-aining that was viewed as having most impact on the teachere’ professional 
development was the local distinct tiaining, when the distinct brought in “experts” in the field of 
bilingual education or other cuinnculum areas. District staff development provided by the teachers’ 
distinct staff was rated as second highest, followed by univeraity preparation [Table X] 

34. Teacher preparation, staff training and administrative support were ranked by teachera as the 
three top factors that continbuted to LEP student success [Table XI] 

35. The success of the seven Study sites can be attinbuted to teacher and principal characteristics/ 
experience, campus and district leadership and support, and focused staff development. Value- 
added characteristics that continbuted to enhancement of progiams for LEP students include 
appropriate assessment features, effective instructional and implementation practices, and 
comprehensive parent involvement 



TABLE m 

ETHNIC BREAKDOWN OF NEW ‘ENROLLMENT (1993-94 TO 1997-98) 



Ethnicity' 


Gains in 
New 

Enrollment 


Representative 
Percentage of 
Total Gains 


White 


32,332 


(11%) 


Hispanic 


195,564 


(67%) 


Afi-ican American 


44,999 


(16%) 


Asian/Native American 


17,143 


(06%) 


TOTAL 


290,038 


(100%) 



Source: TE.A; Public Education Information Management System [PEIMS] 



Table III shows that White students represented 11 percent of new student enrollment (290,038) 
fi'om 1993-94 to 1997-98; Hispanics accounted for 67 percent of new student growth; Afi ican 
American students lepresented 16 percent of the increase, and Asian and Native American 









enrollments combined represented 6 percent of the growth. When considering the demogi'aphic 
characteristics of the student data presented in Table III, it should be noted that minority students 
represented 89 percent of all new student gi'owth in Texas public schools over the four-year period 
analyzed, while non-minority students represented 11 percent of new enrollment. 



TABLE IV: 

GEOGRAPHIC CONCENTRATION OF 
LEP POPULATION BY EDUCATION SERVICE CENTER 



Region 


1991-92 


1997-98 


Plus or 
(Minus) 


Percentage 


‘Percentage 
of State 
Growth 


1 Edinburg 


•*99,918 


115,842 


15,924 


16 


10.000 


2 Corpus Christi 


8,612 


6,608 


(2,004) 


(23) 


N/A 


3 Victoria 


1,777 


2,696 


919 


52 


.006 


4 Houston 


”80,516 


129,715 


49,199 


61 


31.000 


5 Beaumont 


1,531 


2,567 


1,036 


68 


.007 


6 Huntsville 


2,927 


6,674 


3,747 


128 


.020 


7 Kilgore 


3,235 


8,094 


4,859 


150 


.030 


8 Mt. Pleasant 


785 


2,075 


1,290 


, 164 


.008 


9 Wichita Falls 


489 


1,142 


653 


134 


.004 


1 0 Richardson 


”40,344 


78,335 . 


37,991 


94 


24.000 


11 Ft. Worth 


”15,643 


32,463 


16,820 


107 


11.000 


12 Waco 


2,253 


5,853 


3,600 


160 


.020 


1 3 Austin 


11,144 


18,092 


6,948 


62 


.040 


14 Abilene 


1,301 


1,465 


164 


12 


.001 


15 San Angelo 


3,008 


. 3,695 


687 


23 


.004 


16 Amarillo 


4,202 


6,599 


2,397 


57 


.015 


1 7 Lubbock 


4,950 


4,873 


(77) 


(.015) 


N/A 


18 Midland 


8,806 


10,515 


1,709 


19 


.011 


19 El Paso 


”36,932 


48,267 


11,335 


31 


.070 


20 San Antonio 


”32,754 


34,351 


1,597 


.05 


.010 


TOTALS 


361,127 


519,921 


158,794 


*”44 


100.000 



Source: TE.APEIMS Fall 1991-92—1997-98 
• Percentages may not equal 100 due to rounding 

” Regions with higliest concentration of LEP students (306,107) equal to 85 percent of state total ol 361.127 in 1991 >92 
Although total state growth was 44 percent overall, these same regions maintained the highest concentration of LEP 
students as they continued to enroll 84 percent of all new LEP students 
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TABLE V 



GRADE SPAN DISTRIBUTION OF 
LEP STUDENT POPULATION IN TEXAS 



Grade 

Spans 


Enrollment 

1991-92 1997-98 


Numbers and Percentages 
of Increases 


A B 


C 


5 


Total# of LEP 
Students 


Total# of LEP 
Students 


Total LEP 
Student Increases 


*% of Total LEP 
Student Increases 


EE-EL/PreK-5 


249,147 


371,673 


+122,526 


77% 


MS 6-8 


57,301 


81,729 


+24,428 


15 


HS 9-12 


54,679 


66,519 


+11,840 


8% 


TOTALS 


361,127 


519,921 


+158,794 


100% 



Source: TEA POCKET EDITION 1991-92—97-98: PEIMS Fall Data 1991-92—97-98 
Percentage is calculated by dividing total for each grade span in column (C) by total for all grade spans ( 1 58,794) in colunm (C) 



TABLE VI 

TEACHER CHARACTERISTICS 



Yes 88 94.6% 

No 1 .1% 

Uncertain 1 1.1% 

Missing 3 3.2% 

Total 93 100% 


3. lam trained in bilingual methods and 
materials 1 ) Uncertain 2) No, or 3) Yes 


Yes 76 81.7% 

No 6 6.5% 

Uncertain 4 4.3% 

Missing 7 7.5% 

Total 93 100% 


4. I am trained in language assessment: 
1) Uncertain 2) No, or 3) Yes 


Yes 91 97.9% 

No 0 n/a 

Uncertain 0 n/a 

Missing 2 2.2% 

Total 93 100% 


5. I understand the benefits of second 
language learning for limited English 
Proficient students: 1) Uncertain 
2) No, or 3) Yes 


Yes 86 92.5% 

No 2 2.2% 

Uncertain 1 1.1% 

Missing 4 4.3% 

Total 93 100% 


6. lam confident in my training to address 
the needs of limited English proficient 
students: 1 ) Uncertain 2) No, or 3) Yes 
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TABLE Vn 



ETHNIC BREAKDOWN OF NEW TEACHER INCREASES IN TEXAS 

(1993-94 TO 1997-98) 



Ethnicity 


Gains in New Hires 


Representative 
Percentage of Total Gain 


White 


17,190 


50% 


Hispanic 


7,726 


28% 


Afi'ican American 


2,100 


8% 


Asian/Native American 


981 


4% 


TOTAL 


27,997 


100% 



TABLE VIII 

COMPARISON OF NEW STUDENT ENROLLMENT 
TO NEW TEACHER INCREASES IN TEXAS 



Ethnicity 


New Enroliement 
Total for 1997-98 


New Teacher Totals 
1997-98 


Difference 

Shortage/Surplus 


White 


32,332 


(11%) 


17,190 


(61%) 


+55% 


Hispanic 


195,564 


(67%) 


7,726 


(28%) 


-39% 


African American 


44,999 


(16%) 


2,100 


(8%) 


-8% 


Asian/Native American 


17,143 


(0.6%) 


981 


(4%) 


+3.4% 


TOTALS 


290,038 


27,997 


N/A 
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TABLE IX 

PROFILE OF CAMPUS ADMINISTRATORS AT STUDY SITES 



Campus 


Highest 


Certification 


Yrs. in 


Yrs. in 


Yrs. in 


Yrs. 


Vrs. 




Degree 




Prof. Ed. 


Adm. 


Bilingual 


Teaching 


Experience 












Adm. 


Experience 


in Teaching 
Bilingual 
















Ed. 


Bowie 


Masters + 


^Mid-Mgmt. 


20+ 


15-19 


15-19 


5-9 


5-9 






Bilingual 
Elem. Ed. 


years 


years 


years 


years 


years 


Campestre 


Masters + 


Mid-Mgmt. 


15-19 


5-9 


5-9 


10-14 . 


5-9 






Bilingual 
Elem. Ed. 
Secondary 
Ed. 

(Spanish) 


years 


years 


years 


years 


years 


Castaneda 


Masters + 


Mid-Mgmt. 


20+ 


10-14 


10-14 


15-19 


10-14 






Bilingual 
Elem. Ed. 


years 


years 


years 


years 


years 


Clover 


Masters + 


Mid-Mgmt. 


20+ 


5-9 


5-9 


20+ 


15-19 






Bilingual 
Elem. Ed. 
Early Child. 


years 


years 


years 


years 


years 


LaEncantada 


i 

! Masters + 


Mid-Mgmt. 


20+ 


5-9 


5-9 


10-14 


10-14 






Bilingual 
Elem. Ed. 


years 


years 


years 


years 


years 


Kelly 


Masters + 


Mid-Mgmt. 


20+ 


10-14 


10-14 


20+ 


10-14 






Bilingual 
Elem. Ed. 


years 


years 


years 


years 


years 


Scott 


Masters + 


Mid-Mgmt. 


20+ 


20 + 


20+ 


5-9 


20+ 






Bilingual 
Reading 
Elem. Ed. 
Sp. Ed. 
(Spanish) 


years 


years 


years 


years 


years 





‘Mid-Management 










TABLE X 



RANK AND ORDER OF 

PROFESSIONAL DEVELOPMENT OPPORTUNITIES 



Response 


1 


2 


3 


B 


5 


6 


■ 


8 


9 


Cumulative 

Total 


Local Staff 
Development 
Contiact 
Experts 


16 


31 


8 


13 


11 


0 


2 


0 


0 


587 pts. 


Staff 

Development 
Distinct Staff 


19 


19 


16 


13 


7 


2 


0 


1 


0 


558 pts. 


Univ.Teacher 

Preparation 


27 


8 


14 


7 


10 


7 


2 


2 


4 


539 pts. 


ESC-Staff 


4 


17 


20 


22 


12 


4 


1 


1 


1 


526 pts. 


ESC-Contiact 

Experts 


11 


7 


18 


15 


20 


1 


0 


1 


1 


438 pts. 


Staff 

Dev.elopment 

Professional 

Organizations 


1 


4 


2 


0 


13 


30 


4 


2 


6 


262 pts. 


Other 

Annual 

State 

Conferences 


2 


1 


2 


4 


3 


1 


14 


29 


1 


184 pts. 


TEA- 

Sponsored 

State 

Conferences 


0 


2 


5 


0 


1 


7 


20 


6 


9 


165 pts. 


National 

Conferences 


0 


, 3 


0 


0 


2 


4 


2 


11 


31 


109 pts. 
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TABLE XI 



FACTORS CONTRIBUTING TO LEP STUDENTS’ 
SUCCESS/RANKING OUTCOMES 







Second 

Place 




Fourth 

Place 




Total 

Points 


Teacher Preparation 


29 


30 


14 


7 


2 


323 


Staff Training 


31 


19 


10 


9 


4 


283 


Administrative 

Support 


9 


2 


15 


13 


18 


142 


Parental Involvement 


5 


4 


5 


16 


16 


104 


Grouping for 
Insti*uctional Puiposes 


7 


3 


8 


4 


6 


85 


Latitude/ Empowered 
In Teaching 


2 


10 


0 


7 


8 


72 


Materials and 
Resources 






9 


14 


5 


60 


Team Teaching 


2 


3 


4 


2 


3 


41 


Structured Schedule 






10 


3 


5 ■ 


41 


Technology 






5 


2 


11 


30 


Team Planning 


1 


1 


2 


2 


0 


19 


Ongoing Feedback to 
Students/Parents 








2 


2 


6 


Assessment 








1 


1 


3 
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IV. EVALUATION OF THE STUDY 



P art of the Study design allows for the use of an evaluation questionnaire to elicit feedback on 
a continuous basis, as may be provided by educators and non-educators, who read or use the 
Study on an intrastate and interstate basis. A copy of the evaluation form is found at the end 
of the Study to be returned to TEA by one of several means as described herein. 

The evaluation section, copy of which can be found in Appendix G of this document, consists of 
three parts. The first part addresses respondents’ residence within and outside of Texas. Part I 
solicits information fi om respondents regarding types of employment, or whether respondents are 
parents, guardians, and retirees. Part II has been developed to collect data regarding years of 
experience respondents had in various sectore and levels of education. The six categorical response 
modes range from zero (0) years to over twenty years (20+) of experience. Part III is divided into 
five subsections. All respondents are requested to complete the first subsection of Part III. This 
subsection asks five (5) questions to ascertain respondents’ opinions of the Study. The ten questions 
in the second subsection are intended only for school district administiators (Central Administiation), 
which are designed to collect the opinions and views of the Study fiom these respondents. The 
twelve ( 12) questions of the third section are similar to those in the two preceding subsections, but 
are intended only for school district campus administrators. These three sections provide a polytomous 
mode of response (Yes/ No/ No Opinion). The fourth subsection of Part III, allows the school 
district and campus administi ators to rank the contents of the Study in order of priority, from one 
(1) as the highest, to ten (10) as the lowest rank. The fifth subsection, which uses an open-ended 
response mode, elicits optional additional comments fiom all respondents. 

Additionally, an interactive fonnat of the evaluation form was developed and placed on the TEA 
website as part of the Study PDF format. Both the Study and Interactive Evaluation can be 
downloaded fi'om the website by going to http://www.tea.state.tx.us/tsss . This interactive forniat is 
described below. 

The Interactive Evaluation forai is an innovative approach in obtaining feedback about the Study. 
The question and response mode of the form consists of the same three major parts found in the 
published copy of the Study. The responses can be entered on the interactive web page, or mailed in 
as a hard copy of a downloaded version of the evaluation form. Upon accessing the evaluation 
forni, respondents are then guided through a series of screens. As respondents answer questions 
that may apply in the different parts, they are then prompted fomard to the next screen until the 
evaluation is completed. The screens are respondent-sensitive so that particular sectors, such as 
teachers, school district, or campus administrators are prompted to their respective subsections. 
Evaluation data received are quantified, together with those responses fi om mailed questionnaires 
on an ongoing basis. It is anticipated that an annual summai y report will be published by the Program 
Evaluation Unit. The summary report will be based on the evaluation responses obtained from 
persons who have utilized and benefited from the Study report. 
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Through the Agency’s webtrend reporting system, the Program Evaluation Unit is able to obtain 
periodic reports to monitor the number of times that the Study is accessed and downloaded via 
the web site. Since the posting of the Study on the TEA web site in September 6, 2000, reports 
indicate that the Study has had over eleven thousand (11,000) downloads of the entire Study 
document of 2 1 2 pages. 
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APPENDIX A 



STUDY QUESTIONNAIRES 
AND INTERVIEWS 
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Texas Education Agency 
Successful Schools Study 

TEACHER QUESTIONNAIRE 



Using a #2 pencil only, fill-in each oval completely. Please complete the Successful Schools 
Suivey Forai using the various Likert Scales found in the various sections of the form. 

1 . My highest educational level is best described as (Mark only one selection): 

1 = Non-degi'eed 

2 = Associate 

3 = Bachelor 

4 - Masters 

5 = Masters Plus Additional Hours 

6 = Doctorate 

2. I am assigned to the following grade level (Mark only one selection): 

PK 

K 

1 St gi'ade 
2nd grade 
3rd grade 
4th grade 
5 th grade 

3. Total number of professional years in Education: 

Less than 5 yi's. 

5-9 yi's. 

10-14 yi's. 

15-19 yi's. 

20+yrs. 

4. How many years have you taught in Bilingual Education? 

Less than 5 yi's. 

5-9 yi's. 

10-14 yi's. 

15-19yrs. 

20+yrs. 

5. What type of class/stnicture did you teach in during the 95-96 school year (Select only one)? 
Self-contained 

Departmentalized 

Resource 

Multi-age 

Other 

6. What type of class/stnicture did you teach in during the 96-97 school year (Select only one)? 
Self-contained 

Departmentalized 

Resource 

Multi-age 

Other 
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7. What type of class/ stnicture did you teach in during the 97-98 school year? (Select only one) 
Self-contained 

Departmentalized 

Resource 

Multi-age 

Other 

8. How many of your classes involved limited English proficient (LEP) classes? 

All 

Three-fourths 

Half 

One-fourth 

None 

9. What is your proficiency level in Spanish? 

Very Fluent 

Fluent 
Average 
Below Average 
No fluency 

The following items (10-21) are to be answered with Y=Yes, N=No, or U=Uncertain 

10. 1 am assigned to a bilingual education class. 

1 1 . 1 was assigned to Bilingual Education during the 95-96 school year. 

12. 1 was assigned to Bilingual Education during the 96-97 school year. 

13. 1 was assigned to Bilingual Education during the 97-98 school year. 

14. 1 possess a Bilingual Teacher Certificate. 

15.1 possess an ESL Teacher Certificate. 

16. 1 possess an Elementary Teacher Certificate. 

17. 1 possess a Secondary Teacher Certificate. 

1 8. 1 possess a Supervisor Certificate. 

19. 1 possess a Mid-Management Certificate. 

20. 1 possess a Diagnostician Certificate. 

21.1 possess a Certificate that is not mentioned in this suiwey. 

22. My ethnic background is 
Hispanic 

Caucasian (Non-Hispanic) 

African American 
Other 
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23 . My gender is M = Male F = Female 



Please fill in the oval completely using the following: Y=Yes N=No U=Uncertain 

24. 1 am trained in bilingual methods and materials. 

25. 1 am trained in language assessment. 

26. 1 understand the benefits of second language learning for limited English proficient students. 

27. Parents of limited English proficient students at our campus understand the benefits of our 
special programs. 

28. 1 used Spanish most of the time to teach my limited English proficient students. 

29. 1 grouped my limited English proficient students for Spanish according to language 
proficiency in their primary language (Spanish). 

, 30. 1 grouped my limited English proficient students for English instruction according to 
language proficiency in their second (English) 

31. Limited English proficient students in my classes were allowed to express themselves in their 
primary language (Spanish) during teacher and group interaction. 

32. Limited English proficient students in my classes were encouraged to express themselves in 
their second language (English) during teacher and group interaction. 

33. 1 assessed the levels of both primary language (Spanish) and English to ensure appropriate 
instructional focus. 

34. The language levels of my limited English proficient students were assessed on an ongoing 
basis during the school year. 

35. Upon receiving new information from the ongoing language assessments, I modified my 
instruction and placement of my limited English proficient students. 

36. The academic levels of my limited English proficient students were assessed on an ongoing 
basis during the school year. 

37. 1 introduced concepts in the primary language (Spanish) and extended or enriched in English. 

38. 1 am confident in my training to address the needs of limited English proficient students. 

39. 1 was trained through a university/college teacher-training progr am that prepared teachers to 
work with the limited English proficient student population 

40. 1 was trained primarily through staff development and in-serwice to work with the limited 
English proficient population. 

41. My principal provided adequate support for my limited English proficient students. 

42. The District leadership provided adequate supporl for my limited English proficient students. 

43. 1 participated in program decision-making affecting my limited English proficient students. 
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Using the following scale, please indicate the level of implementation of each of these sti ategies 

as follows: 

A=A11 of the Time; M=Most of the Time, S=Some of the Time; R=Rarely, N=Never 

46. 1 provide second language instruction, which develops understanding, speaking, reading, and 
writing skills in English. 

47. 1 provide language arts in Spanish, which includes understanding, speaking, reading, and 
writing skills. 

48. 1 provide instioiction in Spanish in math, science, social studies, and health. 

49. 1 include the teaching of culture in all aspects of the instructional program. 

50. 1 have a system to provide English instioiction to the students with varying levels of language 
proficiency and academic experience. 

51.1 have a system to provide Spanish instmction to the students with varying levels of language 
proficiency and academic experience. 

52. 1 group students according to Spanish language ability for Spanish language arts insti-uction. 

53. 1 group students according to English language ability for English language arts instrtiction. 

54. 1 assess the students’ oral and written proficiency in English on an ongoing basis. 

55. 1 assess the students’ oral and written proficiency in Spanish on an ongoing basis. 

56. 1 have a classroom environment that reflects the students’ culture and learning in two 
languages. 

57. 1 have meaningful parent participation in my class. 

58. 1 am aware of my students’ English language ability early in the school year. 

59. 1 am aware of my students’ Spanish language ability early in the school year. 

60. 1 have clear time allotments for time on task for the content to be taught in English. 

61.1 have clear time allotments for time on task for the content to be taught in Spanish. 

62. 1 encourage my students to take responsibility for their own class work. 

63. 1 prepare my students for lessons by reviewing, outlining, explaining objectives, and 
summarizing. 

64. 1 adjust my teaching pace according to the students’ perceived needs. 

65. 1 am positive, optimistic, and have high expectations of my students. 
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Please answer the following 

1 . Of those listed below which tr aining opportunities/participation has contributed the most to 
your professional development to teach limited English proficient students? Please Rank and 
Order starting with 1,2,3, all the way to 9 with number I being the one you feel has most 
effectively contributed to your professional growth. 

university tr aining as part of the teacher preparation program 

local in-service /staff development provided by school district staff 

local in-serwice/staff development provided by “experts in the field” under contract 

staff development offered by staff of your respective education service centers 

staff development offered by your respective service center using “experts in the 

field” under contract 

staff development offered by professional organizations in the ar ea. List organization (s) 



state conferences: co-sponsored by the Texas Education Agency, such as Migr-ant, 

ACET, etc 

other annual state conference(s) List conference(s) 

other national conference(s) List conference(s) 

2. What five things contr ibuted the most to the academic success of your limited English 
proficient students? (Examples: staff tr aining, teacher preparation, materials latitude in 
teaching, strarctured schedule, parent involvement, administr ative support, grouping for 
instruct ional purposes, team-teaching, instrarctional technology, resources, other. Please rank 
them in order by what you consider as most important first, second important next, and so 
on.) 

1. 

2 . 

3. 

4. 

5. ^ 

Please answer the following. (If you need more space, continue on the back of the sheet. 
Remember to identify each response on the back with appropriate question number) 

1 . What progr am(s) at your campus contributed to the academic success of your limited English 
proficient students? 

2. What approaches/practices have you utilized to ensure language development of your limited 
English proficient students? 

3. a) What approaches/practices have you utilized to ensure the academic success of your 

limited English proficient students? 

b) What specific training have you received that has contributed to your professional 
development to impact on the academic development of the limited English proficient 
students in your classroom? 
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4. a) How long did your limited English students remain in the program (s) that contributed to 

their academic success? (One year, two years, three years, etc.) 

b) Why? 

5. What can your education seiwice center and the Texas Education Agency Do to help 
educators and adminisfrators do an even better job of educating the language minority 
populations in Texas? 

6. What are the most effective ways parents worked with the limited English Proficient students 
at your campus? 

7. How did you utilize parents with your limited English proficient students in your classroom? 

8. In what ways did your campus administration contribute to the success of your Limited 
English proficient students? 

Please feel fi'ee to use this space for any additional comments: 



44 46 

o 

ERIC 



PRINCIPAL QUESTIONNAIRE 



Instructions to all respondents: Please respond to each of the questions and items indicated 
below. Please be assured that the identity of individual principals will be held in confidentiality. 
Your responses are not based on the current school year . Your responses are to be based on 
the 1996-97 school year . Please complete this questionnaire, place it in the envelope upon 
completion and seal it. Amember of TEA’S Study team will visit on-site on the designated date 
to assist you in the completion of this document and collect the inforaiation. Your professional 
contributions and time devoted to the Study are greatly appreciated. Thank you for yOur efforts. 



Narne ^ ^ District 

Campus Name: 

Current Position: 

Ethnicity: Gender: Male Female 



Education Level: Bachelor Master Mastei^- Doctorate 

Certification: Administi-ation (Please list ceitification(s)): 



Bilingual Education 



Other (Please list:) 



Total number of professional years in Education: 
Total number of yeai'S in Administi-ation: 
Administi-ative experience in Bilingual Education: 
Administi-ative experience at this campus: 

Total number of years of Teaching experience: 
Teaching experience in Bilingual Education: 



less than 5 yi'S.. 
less than 5 yi's.. 
less than 5 yi-s.. 
less than 5 yi-s.. 
less than 5 yi-s.. 
less than 5 yi's.. 



5-9 _ 10-14 
5-9_ 10-14 
5-9_ 10-14 
5-9_ 10-14 
5-9_ 10-14 
5-9_ 10-14 



15-19_ 


20+ 


15-19_ 


20+ 


15-19_ 


20+ 


15-19_ 


20+ 


15-19_ 


20+ 


15-19 


20+ 



Please answer the following (If you need more space, continue on the back of the sheet. Please 
remember to identify each response on the back with the appropriate question number). 



1 . What program(s) conti-ibuted to the academic success of the limited English proficient (LEP) 
students in your campus? 

2. What made your teachers successful with their LEP students? 

3. What approaches/practices have you utilized through site-based decision-making that contributed 
to the success of your limited English proficient students? 
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4. a) What specific tiaining have you received that has contributed to your professional 

development to impact on the language development of the language-minority population 
in your classroom? 

b) What specific training have you received that has conti ibuted to your professional 
development to impact on the academic development of the language-minority 
population in your classroom? 

5 a) How long did your LEP students remain in the progiam(s) that contiibuted to their 
academic success? 

b) Why? 

6. What can your education seiwice center and the Texas Education Agency do to help you and 
your staff do an even better job of educating your LEP students? 

7. What are the most effective ways parents worked with the limited English proficient students 
in your campus? 

8. Please share your views on the use of both the first language, Spanish, and the second 
language, English, in class settings: 

Please answer the following: 

1 . Of those listed below, which training opportunities/participation has contiibuted the most to 
your professional development as a campus principal in charge of limited English proficient 
students? Please Rank and Order starting with 1, 2, 3, etc. Rank and Order only those that 
you feel were effective in contiibuting to your professional giowth: 

university training as part of the mid-management/administiator preparation progiam 

local in-seiwice staff development provided by school district staff 

local in-seiwice staff development provided by “experts in the field” under contact 

with the school district 

staff development offered by staff of your respective education seiwice center 

staff development offered by your respective education service center using “experts 

in the field” under confract with the education seiwice center 

staff development offered by professional associations in the area List association(s) 

state conferences co-sponsored by the Texas Education Agency, such as Migrant, 

ACET, TEPSA, TASA, etc. 

annual state conference(s) List conference(s) 

annual national conference(s) List conference(s) 
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1 . Of those listed below, which training oppoitunities/paiticipation have contiibuted the most to 
the professional development of teachers working with limited English proficient students? 
Please Rank and Order starting with 1, 2, 3, etc. Rank and Order only those that you feel 
have effectively contributed to their professional growth: 

university training as part of the teacher preparation program 

local in-service staff development provided by school distinct staff 

local in-service staff development provided by “experts in the field” under contract 

staff development offered by staff of your respective education seiwice center 

staff development offered by your respective education service center using “experts 

in the field” under conrtact 

staff development offered by professional association(s) in the area 

state conferences co-sponsored by the Texas Education Agency, such as Migi'ant, 

ACET, TEPSA. TASA,etc. 

annual state conference(s) List conference(s) 

annual national conference(s) List conference(s) 

3. What five things conti'ibuted the most to the academic success of limited English proficient 
students on your campus? (Examples: staff training, teacher preparation, materials, latitude in 
teaching, parent involvement, instixictional technology, gi'ouping for instmctional puiposes, 
administi'ative support, resources, other. Please rank them in order by what you consider as 
most important first, second important next, and third as last.) 

1. 

2 . ^ 

3 

4. 

5. ^ ' 

Please feel fi'ee to use this space for any additional comments: 
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Texas Education Agency 



SUCCESSFUL SCHOOLS STUDY 1999 
PARENT FOCUS GROUP INTERVIEW 



1 . Please help us identify why children find academic success in this school. What do you do at 
home to contribute your children’s success? 

Por favor ayudenos a identificar las razones porque los ninos de esta escuela han encontiado 
buen exito, academico. <^Que hacen en casa con sus ninos para contiibuir al exito academico? 

2. What is it that the school does to contribute to your children’s success? 

<;.Que es lo que hace la escuela para contiibuir al exito academico de sus ninos? 

3. What is it that the principal does to contribute your children’s success? 

^Que es lo que hace el director de la escuela para contiibuir al exito academico de sus ninos? 

4. What is it that the teacher does to contiibute your children’s success? 

^•.Que es lo que hace la maestra de su nino para contiibuir al exito academico de sus ninos? 

5. What is it that the bilingual program does to contribute your children’s success? 

^Que es lo que hace el progiama bilingiie de la escuela para contiibuir al exito academico de 
sus ninos? 

6. Do you have anything else to add? 

Hay algo mas que quieran comentar? 
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Texas Education Agency 



SUCCESSFUL SCHOOLS STUDY 1999 
DISTRICT INTERVIEW 

1 . What progi am(s) conti ibutes to the academic success of the limited English proficient (LEP) 
students at this campus? 

2. What has made the teachers successful with their LEP students? 

3. What specific training has the distiict provided that has contributed to the professional 
development to impact on the language development of the language minority population in 
the district? 

4. What specific tiaining has the district provided that has contributed to the professional 
development to impact on the academic development of the language minority population in 
the distinct? 

5. How long do the LEP students remain in the progiam(s) that has contributed to their 
academic success? 

Why? 

6. What can your education service center and the Texas Education Agency do to help you and 
your staff do an even better job of educating your LEP students? 

7. What are the most effective ways parents worked with the limited English students at this 
campus? 



Please share your views on the use of the first language, Spanish, and the second language, 
English, in class settings. 
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FORMULAS FOR DESCRIPTIVE STATISTICS 



The nanative of the Study and this manual adopted a general style in order to communicate to 
diverse audiences. Along these lines, the intent of this section is to communicate the nature of 
fomiulas utilized in the descriptive statistical applications to the non-research audience (MacColl 
and White 1998). These fonnulas are divided into two gioupings: 1 ) Measures of Central Tendency, 
and 2) Measures of Variation, which have been discussed in the section on Quantitative Approaches. 
Sources for this section include but have not been limited to Jaeger, (1990), Glass and Hopkins 
( 1984), and Vogt ( 1 993), among othere. Fonnulae for the limited post-hoc analyses, including non- 
parametric approaches, will be featured in subsequent studies as may be conducted. 

Measures of Central Tendency 

> Mean: The arithnietic mean is generally refened to as the average; When the tenn “mean” 
occurs without a modifier, it is assumed to refers to the arithmetic mean. The mean is the sum 
of all values which are then divided by the number of values. The fonnula for the total 
population mean is: 

)Li. = (Zx)/n 

where; 

)Ll = population mean 

Sx = sum of values 
n = number of values 

If the scores are fiom a sample, the formula becomes: 



M = (Xx)/n 

where: 

X = sum of values 
n = number of values. 

The fonnula for M is the same as the foimula for |i,. 

> Median: The median is considered the middlemost value when all the obseiwed values are 
ananged in numerical ascending order or descending. As mentioned earlier, median is a better 
measure of central tendency than the mean when there are standard deviations in the data set. 



53 

er|c 



53 



Based on the foimula from the University of Alberta Instructor Designed Questionnaire 
Systeni{l999), which was refined from the model developed by the University of Michigan, the 
formula then becomes: 



Median = L + 1 * N/2 - F 

/ 



where: 

L = lower limit of the interval containing the median 
I = width of the interval containing the median 
N = total number of respondents 

F = cumulative fr equency conesponding to the lower limit 
/ = number of cases in the interval containing the median 

An alternative fomiulation is: 



Median = L + (^L) (n - C^f) 

f 2 



where 

L = beginning of the interval for the range that contains the median value 
= interval/range length( ^ = range high - range low + 1 ) 

/= number of cases in the inteiwal containing the median 
n = total obseiwations for the table(sum of frequencies) 

Cy/ = cumulative fr equency of the ranges to the lower limit 



^ Mode: The mode is the obseiwed value that occurs most often in the data set (i.e. the value with 
the highest ft'equency). It is often used to estimate which specific value is most likely to occur 
in the future. However, a data set may have more than one mode. A data set is described as: 

♦5* Unimodal if it has only one mode 

*1* Bimodal if it has two modes 

❖ Multiniodal if it has more than two modes {trimodal. quadramodal, etc). 
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Below is an example of a bimodal distribution: 




Measures of Variation 

❖ Standard Deviation: Probably the most utilized measure of central tendency, the 
standard deviation is a measure of the spread of values around the mean. It is also 
the most stable of measures of variability. The popidation standard deviation is 
computed as: 




where; 

G = population standard deviation 
£ = the sum 

X = each individual measurement 



[i = the population mean 



N = the population size. 

The sample estimate of the population standard deviation is computed as; 



V n-i 
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where: 

S = sample estimate standard deviation 
X = the sum 

X = each individual measurement 
X = the sample mean 

X - X = the residual for each measurement. 

/I = the sample size. 

An example of a normal cuiwe with standard deviations is illusti-ated below. 

Standard Deviations within Normal Curve 
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PERCENTAGE OF FORMER LEP STUDENTS PASSING GRADE 5 TAAS (1999) 

READING AT THE 70% LEVEL 
(1994 COHORT) 
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PERCENTAGE OF FORMER LEP STUDENTS PASSING GRADE 5 TAAS (1999) 

READING AT THE 70% LEVEL 
(1994 COHORT) 
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PERCENTAGE OF FORMER LEP STUDENTS PASSING GRADE 5 TAAS 
READING ATT THE 70% LEVEL 
BY YEARS IDENTIFIED AS LEP (1994 COHORT) 




Grade 5 Study Campuses 
-»• Grade 5 External Campuses 
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APPENDIX D 



BENEFITS AND 
CHARACTERISTICS OF 
THE MODE DESIGN 



MULTIPLE OPERATIONS: 
FOUR WAYS OF TRIANGULATION 
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MULTIPLE OPERATIONS: 
BENEFITS FOR THE STUDY 
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APPENDIX E 



RESEARCH QUESTIONS 
AND DATA SOURCES 
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RESARCH QUESTIONS ADDRESSED BY 
MJOR SOURCES OF DATA AND INFORMATION 



Questions 


Interviews 


Surveys 


Site Visits 


Campus & 
Student 
TAAS Data 


AEIS 

Campus 

Report 


What are the LEP, former 
LEP and Non-LEP students' 
academic performance 
as measured by state 
(Grades 3-5) assessments? 








✓ 


✓ 


What are the district leadership 
practices that facilitate 
academic and linguistic 
growth/success for language 
minority students? 


✓ 


✓ 


✓ 




V 


What are the campus 
leadership practices that 
facilitate academic and 
linguistic growtlVsuccess for 1 
anguage minority students? 


✓ 


✓ 


✓ 


✓ 


✓ 


What are the characteristics 
of the teaching staff that 
facilitate academic and 
linguistic growth/success for 
language minority students? 


✓ 


✓ 


✓ 




✓ 


What are the effective teaching 
practices that facilitate 
academic and linguistic 
growth/success for language 
minority students? 


✓ 


✓ 


✓ 


✓ 


✓ 


What are the characteristics 
of parents and parental 
involvement on the seven 
campuses? 


✓ 


✓ 


✓ 






What are the characteristics 
of program(s) serving language 
minority students? 


✓ 


✓ 


✓ 




✓ 


What is the relationship 
between campus practices 
and theory?* 


✓ 


✓ 


✓ 


✓ 


✓ 



* Includes secondary inlbnnation from the review of the literature including state policy documents, related readings, and other 
national, state, and local studies. 
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APPENDIX F 



VALIDITIES AND 
RELIABILITIES OF 
STUDY INSTRUMENTS 
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TEACHER QUESTIONNAIRE 



The questionnaire appeared to meet the criteria for a number of tests for validity and reliability. 

These criteria were in line with those recommended by Fowler (1995), Cuba (1981), and Schofield 

(1990), Some of these included: 

❖ Construct Validity in the questionnaire was accomplished in several ways. First, the pattern of 
outcomes were unifomi across sectors of teachers fi'om the seven campuses. Second, the 
outcomes coincided with those derived from other sources adding to convergent validity which 
is a test for construct validity. Third, preliminary post~hoc conelation analyses showed in selected 
items that similar questions resulted in comparable outcomes. Fourth, the conelations also 
indicated that measures that were unrelated to one another were unassociated (this establishes 
discriminant validity, the second test for construct validity). There is no single measure of 
constrtrct validity. Constrtrct validity is based on the accumulation of knowledge about the 
instrtrment and its relationship to other data gathering tools and operations (Fowler, 1995; 
Andrews, 1984) 

❖ Instrumental validity was apparent through the various operational stages of the questionnaire, 
including field testing administrations of the instrirment, collection of data, and their analyses. 
In addition, the responses of the questionnaire were referenced to the same criterion of separately 
administered of valid stmctured interviews. The employment of pattern identification, fracking, 
and matching added to the validity of the questionnaire (Ragin, 1990; Trochim, 1985; Tmdel 
andAntonius, 1991) 

❖ Interterjudge reliability, or interrater, or intewbsetver reliability refere to the degi'ee of consistency 
by obseiwers in their ratings (De Vellis, 1991). The attainment of interjudge reliability reinforced, in 
turn, synchwnic reliability. This was accomplished through detennining the degree of agreement of 
outcomes fiom administered questionnaires. From a data analysis perepective, statistical outcomes 
indicated the degiee of consistent agreement in processed responses. Initial post-hoc analyses 
reaffiiTned this consistency. This was reinforced by a preliminaiy post-hoc con elation analysis which 
showed in selected items that similar questions resulted in comparable outcomes also indicating 
both infra and inter-rater reliabilities 

Table FI in the following page presents an overview of the Study interventions employed that 

confributed to two types of validity and one type of reliability associated with the questionnaires. 
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TABLE n 

STUDY STRATEGIES OF VALIDITY 
AND RELIABILITY FOR QUESTIONNAIRES 



TESTS 


STUDY INTERVENTIONS 


Comtruct Valiclit}' 


Review and analyses of draft Information by multiple investigators 


Identifying uniform patterns of outcomes from questionnaire 


Establishing a chain of evidence and patterned outcomes 


Comparison of questionnaire outcomes with those from interviews 


Evidence of cotiveiyem and discrminam validities via post hoc analyses 


Instrumental validity 


Standardized field testing, data gathering and approaches to analyses 


Pattern identification from administrations of questionnaires 


Matching of patterns and outcomes with those from interviews 


Systematic explanation building based on multiple outcomes 


Interterjudge reliabilit}' 


Identification and matching of patterned outcomes from interviews 


Systematic review of data outcomes by multiple investigators 


Identifying consistent patterns of questionnaire outcomes 


Matching corresponding patterned of outcomes from respondents 


Confirming outcomes by initial post-hoc analyses (inter-/intra«ratings) 



INTERVIEWS 

The incorporation of interviewing as part of a divereity of approaches is in line with research which 
views it as a “validity check” through field research (Webb, 1966; Kirk and Miller, 1986). By way of 
coiTobomting outcomes fi'om other sources of data, inteiwiewing contiibuted to the robustness of the 
design. The collective format of the teacher and inteiwiews allowed for “convereation with puipose” 
(Denzin and Lincoln, 1994; Dexter, 1970 Jacobson, 1988). Several types of validities and reliabilities 
appear to apply to the inteiwiews. The validities, reliabilities, and criteria for interviews are as follow: 

❖ Comtntct Validity for interviews was primarily accomplished by the establishment of coiTectly defined 
and consistently applied protocols for the concepts under study. The application of pattern matching 
contiibuted to increased comtmct validity. This idea of using pattern matching as a rubric for assessing 
constmct validity originated with Trochim (1985, 1989). As with the Teacher Questionnaire, the 
outcomes from inteiwiews coincided with those derived fi'om other sources adding to convei'gent 
validity which is a test for constmct validity, no plausible alternative theories accounted for the 
observed patterns. Cronbach (1971) later ai;gued for an extended concept of constmct validity which 
’’pertains to qualitative summaries as well as numerical scores.... it is an open process where to 
validate is to investigate” (p. 433; also Kvale, 1995). These summaries were augmented by field 
notes and an “audit frail” of processes. Multiple approaches and outcomes from questionnaires and 
case studies reinforced the valid association of these measures with concepts. Multiple investigator 
reviewed chains of evidence produced as outcomes 
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❖ Internal Validity for interviews occuired by applying pattern-seeking and matching methods 
across cases. The consistent application of Delphi-like approaches added to the validity by 
providing for member checks, which brought the outcomes and inteipretations back to 
respondents to determine whether they were plausible. Additional peer examination allowed 
colleagues to comment on findings as they emerged (Denzen, 1970; Meniam, 1998; 1988). 
Explanation building was consistently applied to outcomes (Ader and Mellenberg, 1999; 
Connaway, 1 996). Data were viewed both as a continuity, and as separate cases within fixed 
time periods 

<♦ Interterjudge reliability for interviews was reinforced by Delphi-like approaches which provided a 
mechanism for respondents to ascertain plausibility of outcomes (Enzer, 1971; The Interdisciplinary 
Center Comparative for Research in the Social Sciences, 1999). Interterjudge reliability was 
accomplished through determining the degiee of consistent agieement from analyzed responses. 
These outcomes were reviewed by a number of investigatoi's who reinforced the reliability of outcomes 
through their concunence (Creswell, 1994; Gazel, et. al., 1998; Bloland, 1992; Fettennan, 1988). 

Table F2 below presents an oveiview of the Study intervention employed that contributed to two 
types of validity and one type of reliability associated with the conducted interviews. 

TABLE F2 

STUDY STRATEGIES OF VALIDITY AND RELIABILITY FOR INTERVIEWS 



TESTS 


STUDY INTERVENTIONS 


Construct Validity 


Standardized approaches and augmented summaries 


Established chain of evidence and patterned outcomes 


Matching patterned concurrence of outcomes with those from 
questionnaires 


ItUernal VaU(lit\> 


Review and comparisons of outcomes by multiple investigators 


Pattern Matching across and within cases 


Member checks and additional peer evaluation 


Systematic analyses, reviews, and logical explanation building 


luterierjiulge reUabUitX' 


Identification and matching of patterned outcomes from interviews 


Systematic review of data outcomes by multiple investigators 


Establishing consistent patterns of interview outcomes 


Matching corresponding patterned of outcomes from respondents 



CASE STUDIES 

If similar themes are noted in data collection fr om the different sources, the credibility of the 
inteipretations is enhanced. This multiple case study research explored the characteristics of the 
district leadership in providing support for bilingual education at the campus level for the campus 
administrators, for the teachers, and for the parents whose limited Englishproficient students attended 
the seven successful schools. Yin (1994), proposed a schematic for deteiTnining three (3) validities 
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and a general definition of reliability of applied case studies. These criteria were adopted as shown 
in its modified form below. Essentially, all of these criteria had been met by the Study: 

*1* Construct Validity was primarily accomplished the establishment of coiTectly defined and 
consistently applied operational measures for the concepts under study. Multiple sources of 
evidence encouraged convergent lines of inquiry. Multiple investigators reviewed chains of 
evidence produced as outcomes. Multiple case approaches, in turn, reinforced the valid 
association of these measures with concepts by obseiwed similarities with specific time fi'ames, 
and across campuses 

❖ Internal Validity occuiTed by applying analytical pattern-seeking and matching methods across 
cases. Explanation building was carefully and consistently applied to outcomes. Data were 
viewed both as a continuity and as separate cases within fixed time periods. The convergence of 
multiple sources of evidence to similar degrees across campuses confributed to the internal 
validity of the case studies 

<♦ External Validity is considered the greatest weakness of experimental designs, because 
generalizability does not go much beyond the laboratory setting. Conversely, naturalistic 
approaches can lend themselves to much gi'eater degrees of external validity. The criteria for 
meeting the test of this validity through intei*views was perceived through the use of multiple 
case-studies, and the obseiwed similarities within time-fi'ameworks of outcomes. These outcomes 
lent themselves to a generalizability across cases (Kirk and Miller, 1986; Sadler, 1985). 

♦J* Reliability according to the applied definition fi'om Yin (1994), infers replication of outcomes. 
The MODE design also treated cases as individual entities without contact with one another, 
and over periods of delineated time. As with internal validity. The convergence of multiple 
sources of evidence to similar degrees across campuses contributed to the reliability of the case 
studies. Since patterned outcomes were similar over these periods, it is contended that the 
application of case studies met this test of reliability. 

The study sfrategies utilized for case study validity and reliability based on Yin ( 1 994) ai'e summarized 
in Table F3 below. 



TABLE F3 

STUDY STRATEGIES OF VALIDITY AND RELIABILITY FOR CASE STUDIES 



TESTS 


STUDY INTERVENTIONS 


Construct Valldit}’ 


Use of multiple sources 




Established chain of evidence 




Review of draft information by multiple investigators 


Intenial Validity' 


Pattern matching 




Explanation building 




Analyses over time periods 


Exienial Validity 


Replication logic in multiple case studies 


Reliability! 


Case study protocols 




Case study data base 
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EVALUATION FORM FOR 
THE TEXAS SUCCESSFUL SCHOOLS STUDY 



The Evaluation Fomi for the Study has been included as part of the final report of the uccessfiil 
Schools Study to assist the Progr am Evaluation Unit in obtaining feedback regarding the Study. We 
encourage all persons who review and use the Study to help us by providing feedback in the areas 
listed below. Thank you for taking time to fill out this evaluation fonn and to share your views on 
the Study. Your responses are very important to this agency effort. We intend to utilize your responses 
as we consider possible expansion of the Study and future study efforts. 

All persons who review/read the contents of The Texas Successful Schools Study are encouraged 
to respond to each part of the fonii as per instinrctions noted. Upon completion of the foiTn, please 
mail tt to either: 



Oscar M. Cardenas, Study Administr ator 
or 

Stan Seidner, Program Director 
at the 

Program Evaluation Unit 
Office for the Education of Special Populations 
Texas Education Agency 
1701 N. Congress Avenue 
Austin, Texas 78701-1494 

Or fax the completed foiTn to (512) 463 7441 . 



Please complete the following information for our mailing list: 

Name: Title: 

Mailing address: 

E-Mail address: 
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PART lA: Please respond to the appropriate item below: 



I CURRENTLY RESIDE: IN TEXAS OUTSIDE OF TEXAS 

PLEASE INDICATE CITY AND STATE: 

1 . 1 cunently work at a: oFederal Agency oState Education Agency oOther State Education Agency 

oUniversity oCollege oTechnical School oProprietary School 

oPublic School oPrivate School oCharter School oEducation Service Center 

oOther (Please describe): 

2. 1 am self-employed o 

3. 1 am an elected or appointed official (if applicable) oMunicipal (local) oCounty 

(Please designate governmental level) oState o Federal 

4. 1 am responding as a parent or guardian with a child in school o 

5. 1 am responding as a retired educator o 



PART IB: Please respond to appropriate item below: 

1 . 1 am cunently a/an: o Administrator oCampus Administrator oTeacher 
0 Professor oTeacher aide oConsultant oOther 



PART II: Please indicate your years of experience in education with an X in the appropriate box 
for each listed item. 



YEARS OF EXPERIENCE 


0 


1-4 


5-9 


10-14 


15-19 


204- 


Elementary 














Secondary 














Bilingual Education 














English as a Second Language 














Administration 














•Administration of Bilingual/ESL Programs 














Supervision 














University Training of Bilingual Teachers 














University Training of ESL Teachers 
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PART III; 



All Respondents, please respond with an X to reflect your opinions 



> This type of study was needed 


Yes 


No 


No Opinion 


> The study will be helpful to both 
educators and administrators in 
working with all children 


Yes 


No 


No Opinion 


> The study helped me understand 
why the use of the home language 
is important to children’s success 


Yes 


No 


No Opinion 


> I will recommend the study to 
other persons to support my school 
. distria’s eflForts to achieve success 


Yes 


No 


No Opinion 


> More studies of this type on other 
children with special needs should 
be done by school districts and the 
state education agency. 


Yes 


No 


No Opinion 


School District Administrators only, please respond with X to reflect your opinion 


> This type of study was needed 
state education agency. 


Yes 


No 


No Opinion 


> The study will be helpful to both educators 
and administrators in working with all children 


Yes 


No 


No Opinion 


> This study will be of assistance to me as we 
work to improve the quality of programs for 
students with special needs 


Yes 


No 


No Opinion 


> This study will be of assistance 
to district administrators in assessing 
and improving current practices and programs 


Yes 


No 


No Opinion 


> The report is easy to follow 


Yes 


No 


No Opinion 


> The case studies will be helpful to 
focus on district wide staff training 


Yes 


No 


No Opinion 


> The study or salient features of the report 
should be shared with district statf 


Yes 


No 


No Opinion 


> There are effective practices and 
program features that can be replicated 


Yes 


No 


No Opinion 


^ The agency should conduct other studies 
of this nature to assist school districts in 
achieving both quality and excellence 
in education for all children 


Yes 


. No 


No Opinion 


> The study serves as an effective 
resource guide regarding research 
and approaches to second language learning 


Yes 


No 


No Opinion 
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School District Campus Administrators only, please respond with X to reflect your opinion 



This type of study was needed 


Yes 


No 


No Opinion 


The study will be helpful to both teachers 
and parents in working with all our children 


Yes 


No 


No Opinion 


This study will be of assistance to me as we 
work to improve the quality of programs for 
students with special needs in our campus 


Yes 


No 


No Opinion 


This study will be of assistance to us in 
assessing, redireaing or improving current 
practices and programs 


Yes 


No 


No Opinion 


The report is easy to follow 


Yes 


No 


No Opinion 


The case studies will be helpful to 
focus on campus staff training 


Yes 


No 


No Opinion 


The study or salient features of the report 
will be shared with campus staff 


Yes 


No 


No Opinion 


There are effective practices and program 
features that we are interested in replicating 
or adapting 


Yes 


No 


No Opinion 


The agency should conduct other studies to 
assist school districts in achieving both quality 
and excellence in education for all children 


Yes 


No 


No Opinion 


The study serves as an effective resource guide 
regarding research and approaches to second 
language learning 


Yes 


No 


No Opinion 


In my professional opinion, this study can have 
a great impact on teaching and learning for 
all children 


Yes 


No 


No Opinion 


The study report is too massive to use as a 
resource manual for training and instruction 


Yes 


No 


No Opinion 



SCHOOL DISTRICT AND CAMPUS ADMINISTRATORS ONLY 



Priority Rankings of Study Content 



Please indicate which sections or appendices of the report you think will be most helpful in your present 
capacity by assigning priority rankings (from 1 as highest to 1 0 as lowest) below: 

Executive Summary Scope of the Study, Research Design and Methodology 

Introduction and Background Enrollment and Teacher Statistics 

Need for the Study Staf Characteristics 

Findings Composite Study Results 

Student and Cmpus Student Performance Analysis 

Performance Study Questionnaire 
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Additional Comments you may wish to share 



On behalf of the Office for the Education of Special Populations, we convey our appreciation for sharing your 
time and views with us on this important Statewide Leadership effort and 
other studies that may be conducted 



For questions on the Evaluation Foim or the Successful Schools Study report, please contact: 

Oscar M. Gardena Stan Seidner 

Principal Investigator Program Director 

(512)463-9714 OR (512)475-3489 

ocardena@tmail.tea.state.tx.us sseidner@tmail.tea.state.tx.us 
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COMPLIANCE STATEMENT 



TITLE VI, CIVIL RIGHTS ACT OF 1964; THE MODIFIED COURT ORDER, CIVIL ACTION 5281, 
FEDERAL DISTRICT COURT, EASTERN DISTRICT OF TEXAS, TYLER DIVISION 
Reviews of local education agencies pertaining to compliance with Title VI Civil Rights Act of 1964 and with specific 
requirements of the Modified Court Order, Civil Action No. 5281, Federal District Court, Eastern District of Texas, 
Tyler Division are conducted periodically by staff representatives of the Texas Education Agency. These reviews cover 
at least the following policies and practices: 

(T ) acceptance policies on student transfers from other school districts; 

(2 ) operation of school bus routes or runs on a nonsegregated basis; 

(3 ) nondiscrimination in extracurricular activities and the use of school facilities; 

(4) nondiscriminatory practices in the hiring, assigning, promoting, paying, demoting, reassigning, 
or dismissing of faculty and staff members who work with children; 

(5) enrollment and assignment of students without discrimination on the basis of race, color, or 
national origin; 

(6 ) nondiscriminatory practices relating to the use of a student's first language; and 
( 7 ) evidence of published procedures for hearing complaints and grievances. 

In additionto conducting reviews, the Texas Education Agency staff representatives check complaints of discrimination 
made by a citizen or citizens residing in a school district where it is alleged discriminatory practices have occurred or 
are occurring. 

Where a violation of Title VI of the Civil Rights Act is found, the findings are reported to the Office for Civil Rights, 
U.S. Department of Education. 

If there is a direct violation of the Court Order in Civil Action No. 528 1 that cannot be cleared through negotiation, the 
sanctions required by the Court Order are applied. 

TITLE VH, CIVIL RIGHTS ACT OF 1964 AS AMENDED BY THE EQUAL EMPLOYMENT OPPORTUNITY 
ACT OF 1972; EXECUTIVE ORDERS 11246 AND 11375; EQUALPAY ACT OF 1964; TITLE LX, EDUCATION 
AMENDMENTS; REHABILITATION ACT OF 1973 AS AMENDED; 1974 AMENDMENTS TO THE WAGE- 
HOUR LAW EXPANDING THE AGE DISCRIMINATION IN EMPLOYMENT ACT OF 1967; VIETNAM 
ERA VETERANS READJUSTMENTASSISTANCE ACTOF 1972 AS AMENDED; IMMIGRATION REFORM 
AND CONTROL ACT OF 1986; AMERICANS WITH DISABILITIES ACT OF 1990; AND THE CIVIL 
RIGHTSACTOF1991. 

The Texas Education Agency shall comply fully with the nondiscrimination provisions of all federal and state laws, 
rules, and regulations by assuring that no person shall be excluded from consideration for recruitment, selection, 
appointment, training, promotion, retention, or any other personnel action, or be denied any benefits or participation in 
any educational programs or activities which it operates on the grounds of race, religion, color, national origin, sex, 
disability, age, or veteran status (except where age, sex, or disability constitutes a bona fide occupational qualification 
necessary to proper and efficient administration ). The Texas Education Agency is an Equal Employment Opportunity/ 
Affirmative Action employer. 
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